Upload folder using huggingface_hub

Mistral-7B-Instruct-v0.3_int4_hellaswag-routerbench-0shot_lr-0.0002_e-8_seq-512_lora-a-32-d-0.05-r-64_bs-1_gas-2_tf32-True_tunedata-portion-p-0.2-num-1405-sd-4/README.md

@@ -0,0 +1,202 @@
+---
+base_model: mistralai/Mistral-7B-Instruct-v0.3
+library_name: peft
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.13.1

Mistral-7B-Instruct-v0.3_int4_hellaswag-routerbench-0shot_lr-0.0002_e-8_seq-512_lora-a-32-d-0.05-r-64_bs-1_gas-2_tf32-True_tunedata-portion-p-0.2-num-1405-sd-4/adapter_config.json

@@ -0,0 +1,29 @@
+{
+  "alpha_pattern": {},
+  "auto_mapping": null,
+  "base_model_name_or_path": "mistralai/Mistral-7B-Instruct-v0.3",
+  "bias": "none",
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 32,
+  "lora_dropout": 0.05,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "r": 64,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "q_proj",
+    "v_proj"
+  ],
+  "task_type": "CAUSAL_LM",
+  "use_dora": false,
+  "use_rslora": false
+}

Mistral-7B-Instruct-v0.3_int4_hellaswag-routerbench-0shot_lr-0.0002_e-8_seq-512_lora-a-32-d-0.05-r-64_bs-1_gas-2_tf32-True_tunedata-portion-p-0.2-num-1405-sd-4/adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3d7973ea17ba1344cdd86b7c04f170349bb3df7f8ce81e914e1cd02d9156a5b9
+size 109069176

Mistral-7B-Instruct-v0.3_int4_hellaswag-routerbench-0shot_lr-0.0002_e-8_seq-512_lora-a-32-d-0.05-r-64_bs-1_gas-2_tf32-True_tunedata-portion-p-0.2-num-1405-sd-4/checkpoint-1218/README.md

@@ -0,0 +1,202 @@
+---
+base_model: mistralai/Mistral-7B-Instruct-v0.3
+library_name: peft
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.13.1

Mistral-7B-Instruct-v0.3_int4_hellaswag-routerbench-0shot_lr-0.0002_e-8_seq-512_lora-a-32-d-0.05-r-64_bs-1_gas-2_tf32-True_tunedata-portion-p-0.2-num-1405-sd-4/checkpoint-1218/adapter_config.json

@@ -0,0 +1,29 @@
+{
+  "alpha_pattern": {},
+  "auto_mapping": null,
+  "base_model_name_or_path": "mistralai/Mistral-7B-Instruct-v0.3",
+  "bias": "none",
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 32,
+  "lora_dropout": 0.05,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "r": 64,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "q_proj",
+    "v_proj"
+  ],
+  "task_type": "CAUSAL_LM",
+  "use_dora": false,
+  "use_rslora": false
+}

Mistral-7B-Instruct-v0.3_int4_hellaswag-routerbench-0shot_lr-0.0002_e-8_seq-512_lora-a-32-d-0.05-r-64_bs-1_gas-2_tf32-True_tunedata-portion-p-0.2-num-1405-sd-4/checkpoint-1218/adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:668d34419a8dc0cd4a57cee940c9d0cd769e1f5aaf9c77d140888df2a2fb1a8d
+size 109069176

Mistral-7B-Instruct-v0.3_int4_hellaswag-routerbench-0shot_lr-0.0002_e-8_seq-512_lora-a-32-d-0.05-r-64_bs-1_gas-2_tf32-True_tunedata-portion-p-0.2-num-1405-sd-4/checkpoint-1218/optimizer.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b022d0642cdc44df92079d8aee69bfafa216b0b3db43fd2a04d430396c830538
+size 55532666

Mistral-7B-Instruct-v0.3_int4_hellaswag-routerbench-0shot_lr-0.0002_e-8_seq-512_lora-a-32-d-0.05-r-64_bs-1_gas-2_tf32-True_tunedata-portion-p-0.2-num-1405-sd-4/checkpoint-1218/rng_state.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8750639bf51391a8e903d0d30fabff4adbbe3c3c7fd74052f8cea190c537fa5f
+size 14244

Mistral-7B-Instruct-v0.3_int4_hellaswag-routerbench-0shot_lr-0.0002_e-8_seq-512_lora-a-32-d-0.05-r-64_bs-1_gas-2_tf32-True_tunedata-portion-p-0.2-num-1405-sd-4/checkpoint-1218/scheduler.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:be446f2daea78902e7f0241fb3dc8c52e409750a9dbc64a165213eefd4c40825
+size 1064

Mistral-7B-Instruct-v0.3_int4_hellaswag-routerbench-0shot_lr-0.0002_e-8_seq-512_lora-a-32-d-0.05-r-64_bs-1_gas-2_tf32-True_tunedata-portion-p-0.2-num-1405-sd-4/checkpoint-1218/special_tokens_map.json

@@ -0,0 +1,24 @@
+{
+  "bos_token": {
+    "content": "<s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "</s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": "</s>",
+  "unk_token": {
+    "content": "<unk>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

Mistral-7B-Instruct-v0.3_int4_hellaswag-routerbench-0shot_lr-0.0002_e-8_seq-512_lora-a-32-d-0.05-r-64_bs-1_gas-2_tf32-True_tunedata-portion-p-0.2-num-1405-sd-4/checkpoint-1218/tokenizer.model

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:37f00374dea48658ee8f5d0f21895b9bc55cb0103939607c8185bfd1c6ca1f89
+size 587404

Mistral-7B-Instruct-v0.3_int4_hellaswag-routerbench-0shot_lr-0.0002_e-8_seq-512_lora-a-32-d-0.05-r-64_bs-1_gas-2_tf32-True_tunedata-portion-p-0.2-num-1405-sd-4/checkpoint-1218/trainer_state.json

@@ -0,0 +1,912 @@
+{
+  "best_metric": 1.790574550628662,
+  "best_model_checkpoint": "outputs-001/Mistral-7B-Instruct-v0.3_int4_hellaswag-routerbench-0shot_lr-0.0002_e-8_seq-512_lora-a-32-d-0.05-r-64_bs-1_gas-2_tf32-True_tunedata-portion-p-0.2-num-1405-sd-4/checkpoint-304",
+  "epoch": 4.0,
+  "eval_steps": 10,
+  "global_step": 1218,
+  "is_hyper_param_search": false,
+  "is_local_process_zero": true,
+  "is_world_process_zero": true,
+  "log_history": [
+    {
+      "epoch": 0.03284072249589491,
+      "grad_norm": 0.8926695585250854,
+      "learning_rate": 0.0002,
+      "loss": 2.7246,
+      "step": 10
+    },
+    {
+      "epoch": 0.06568144499178982,
+      "grad_norm": 1.181358814239502,
+      "learning_rate": 0.0002,
+      "loss": 2.1913,
+      "step": 20
+    },
+    {
+      "epoch": 0.09852216748768473,
+      "grad_norm": 0.7392292618751526,
+      "learning_rate": 0.0002,
+      "loss": 2.1022,
+      "step": 30
+    },
+    {
+      "epoch": 0.13136288998357964,
+      "grad_norm": 0.7944067716598511,
+      "learning_rate": 0.0002,
+      "loss": 2.071,
+      "step": 40
+    },
+    {
+      "epoch": 0.16420361247947454,
+      "grad_norm": 0.7858729958534241,
+      "learning_rate": 0.0002,
+      "loss": 2.0564,
+      "step": 50
+    },
+    {
+      "epoch": 0.19704433497536947,
+      "grad_norm": 0.7574328184127808,
+      "learning_rate": 0.0002,
+      "loss": 2.065,
+      "step": 60
+    },
+    {
+      "epoch": 0.22988505747126436,
+      "grad_norm": 0.694824755191803,
+      "learning_rate": 0.0002,
+      "loss": 1.9014,
+      "step": 70
+    },
+    {
+      "epoch": 0.2627257799671593,
+      "grad_norm": 0.6516484022140503,
+      "learning_rate": 0.0002,
+      "loss": 1.885,
+      "step": 80
+    },
+    {
+      "epoch": 0.2955665024630542,
+      "grad_norm": 0.6870642304420471,
+      "learning_rate": 0.0002,
+      "loss": 1.8555,
+      "step": 90
+    },
+    {
+      "epoch": 0.3284072249589491,
+      "grad_norm": 0.7598554491996765,
+      "learning_rate": 0.0002,
+      "loss": 1.84,
+      "step": 100
+    },
+    {
+      "epoch": 0.361247947454844,
+      "grad_norm": 0.6050140857696533,
+      "learning_rate": 0.0002,
+      "loss": 1.8204,
+      "step": 110
+    },
+    {
+      "epoch": 0.39408866995073893,
+      "grad_norm": 0.7373790740966797,
+      "learning_rate": 0.0002,
+      "loss": 1.8294,
+      "step": 120
+    },
+    {
+      "epoch": 0.4269293924466338,
+      "grad_norm": 0.6588838696479797,
+      "learning_rate": 0.0002,
+      "loss": 1.7682,
+      "step": 130
+    },
+    {
+      "epoch": 0.45977011494252873,
+      "grad_norm": 0.5314497947692871,
+      "learning_rate": 0.0002,
+      "loss": 1.7925,
+      "step": 140
+    },
+    {
+      "epoch": 0.49261083743842365,
+      "grad_norm": 0.6273255944252014,
+      "learning_rate": 0.0002,
+      "loss": 1.7992,
+      "step": 150
+    },
+    {
+      "epoch": 0.5254515599343186,
+      "grad_norm": 0.5744606256484985,
+      "learning_rate": 0.0002,
+      "loss": 1.8207,
+      "step": 160
+    },
+    {
+      "epoch": 0.5582922824302134,
+      "grad_norm": 0.577231228351593,
+      "learning_rate": 0.0002,
+      "loss": 1.8215,
+      "step": 170
+    },
+    {
+      "epoch": 0.5911330049261084,
+      "grad_norm": 0.5391182899475098,
+      "learning_rate": 0.0002,
+      "loss": 1.8125,
+      "step": 180
+    },
+    {
+      "epoch": 0.6239737274220033,
+      "grad_norm": 0.5863330364227295,
+      "learning_rate": 0.0002,
+      "loss": 1.8501,
+      "step": 190
+    },
+    {
+      "epoch": 0.6568144499178982,
+      "grad_norm": 0.6266984343528748,
+      "learning_rate": 0.0002,
+      "loss": 1.8055,
+      "step": 200
+    },
+    {
+      "epoch": 0.6896551724137931,
+      "grad_norm": 0.5842767357826233,
+      "learning_rate": 0.0002,
+      "loss": 1.7923,
+      "step": 210
+    },
+    {
+      "epoch": 0.722495894909688,
+      "grad_norm": 0.5255483984947205,
+      "learning_rate": 0.0002,
+      "loss": 1.7906,
+      "step": 220
+    },
+    {
+      "epoch": 0.7553366174055829,
+      "grad_norm": 0.7259632349014282,
+      "learning_rate": 0.0002,
+      "loss": 1.786,
+      "step": 230
+    },
+    {
+      "epoch": 0.7881773399014779,
+      "grad_norm": 0.5593494176864624,
+      "learning_rate": 0.0002,
+      "loss": 1.7291,
+      "step": 240
+    },
+    {
+      "epoch": 0.8210180623973727,
+      "grad_norm": 0.5614203214645386,
+      "learning_rate": 0.0002,
+      "loss": 1.8098,
+      "step": 250
+    },
+    {
+      "epoch": 0.8538587848932676,
+      "grad_norm": 0.6784001588821411,
+      "learning_rate": 0.0002,
+      "loss": 1.814,
+      "step": 260
+    },
+    {
+      "epoch": 0.8866995073891626,
+      "grad_norm": 0.5510041117668152,
+      "learning_rate": 0.0002,
+      "loss": 1.6856,
+      "step": 270
+    },
+    {
+      "epoch": 0.9195402298850575,
+      "grad_norm": 0.5453769564628601,
+      "learning_rate": 0.0002,
+      "loss": 1.7939,
+      "step": 280
+    },
+    {
+      "epoch": 0.9523809523809523,
+      "grad_norm": 0.5494107604026794,
+      "learning_rate": 0.0002,
+      "loss": 1.7489,
+      "step": 290
+    },
+    {
+      "epoch": 0.9852216748768473,
+      "grad_norm": 0.5768024921417236,
+      "learning_rate": 0.0002,
+      "loss": 1.8442,
+      "step": 300
+    },
+    {
+      "epoch": 0.9983579638752053,
+      "eval_loss": 1.790574550628662,
+      "eval_runtime": 147.7829,
+      "eval_samples_per_second": 3.607,
+      "eval_steps_per_second": 0.453,
+      "step": 304
+    },
+    {
+      "epoch": 1.0180623973727423,
+      "grad_norm": 0.5931118130683899,
+      "learning_rate": 0.0002,
+      "loss": 1.7061,
+      "step": 310
+    },
+    {
+      "epoch": 1.0509031198686372,
+      "grad_norm": 0.6433814764022827,
+      "learning_rate": 0.0002,
+      "loss": 1.6997,
+      "step": 320
+    },
+    {
+      "epoch": 1.083743842364532,
+      "grad_norm": 0.6801510453224182,
+      "learning_rate": 0.0002,
+      "loss": 1.697,
+      "step": 330
+    },
+    {
+      "epoch": 1.116584564860427,
+      "grad_norm": 0.6511064171791077,
+      "learning_rate": 0.0002,
+      "loss": 1.721,
+      "step": 340
+    },
+    {
+      "epoch": 1.1494252873563218,
+      "grad_norm": 0.6265677213668823,
+      "learning_rate": 0.0002,
+      "loss": 1.7209,
+      "step": 350
+    },
+    {
+      "epoch": 1.1822660098522166,
+      "grad_norm": 0.6211821436882019,
+      "learning_rate": 0.0002,
+      "loss": 1.6524,
+      "step": 360
+    },
+    {
+      "epoch": 1.2151067323481117,
+      "grad_norm": 0.6931810975074768,
+      "learning_rate": 0.0002,
+      "loss": 1.7091,
+      "step": 370
+    },
+    {
+      "epoch": 1.2479474548440066,
+      "grad_norm": 0.5978460311889648,
+      "learning_rate": 0.0002,
+      "loss": 1.6604,
+      "step": 380
+    },
+    {
+      "epoch": 1.2807881773399015,
+      "grad_norm": 0.6581379175186157,
+      "learning_rate": 0.0002,
+      "loss": 1.6707,
+      "step": 390
+    },
+    {
+      "epoch": 1.3136288998357963,
+      "grad_norm": 0.6794891953468323,
+      "learning_rate": 0.0002,
+      "loss": 1.6543,
+      "step": 400
+    },
+    {
+      "epoch": 1.3464696223316914,
+      "grad_norm": 0.6028420329093933,
+      "learning_rate": 0.0002,
+      "loss": 1.7261,
+      "step": 410
+    },
+    {
+      "epoch": 1.3793103448275863,
+      "grad_norm": 0.8568810820579529,
+      "learning_rate": 0.0002,
+      "loss": 1.7234,
+      "step": 420
+    },
+    {
+      "epoch": 1.4121510673234812,
+      "grad_norm": 0.6654542088508606,
+      "learning_rate": 0.0002,
+      "loss": 1.6885,
+      "step": 430
+    },
+    {
+      "epoch": 1.444991789819376,
+      "grad_norm": 0.6363505721092224,
+      "learning_rate": 0.0002,
+      "loss": 1.7335,
+      "step": 440
+    },
+    {
+      "epoch": 1.477832512315271,
+      "grad_norm": 0.7018159627914429,
+      "learning_rate": 0.0002,
+      "loss": 1.6947,
+      "step": 450
+    },
+    {
+      "epoch": 1.5106732348111658,
+      "grad_norm": 0.6718074679374695,
+      "learning_rate": 0.0002,
+      "loss": 1.7006,
+      "step": 460
+    },
+    {
+      "epoch": 1.5435139573070606,
+      "grad_norm": 0.6456331610679626,
+      "learning_rate": 0.0002,
+      "loss": 1.6504,
+      "step": 470
+    },
+    {
+      "epoch": 1.5763546798029555,
+      "grad_norm": 0.7088801860809326,
+      "learning_rate": 0.0002,
+      "loss": 1.6953,
+      "step": 480
+    },
+    {
+      "epoch": 1.6091954022988506,
+      "grad_norm": 0.8001429438591003,
+      "learning_rate": 0.0002,
+      "loss": 1.6521,
+      "step": 490
+    },
+    {
+      "epoch": 1.6420361247947455,
+      "grad_norm": 0.6954358816146851,
+      "learning_rate": 0.0002,
+      "loss": 1.6384,
+      "step": 500
+    },
+    {
+      "epoch": 1.6748768472906403,
+      "grad_norm": 0.6385519504547119,
+      "learning_rate": 0.0002,
+      "loss": 1.6606,
+      "step": 510
+    },
+    {
+      "epoch": 1.7077175697865354,
+      "grad_norm": 0.6736445426940918,
+      "learning_rate": 0.0002,
+      "loss": 1.7151,
+      "step": 520
+    },
+    {
+      "epoch": 1.7405582922824303,
+      "grad_norm": 0.6794690489768982,
+      "learning_rate": 0.0002,
+      "loss": 1.7363,
+      "step": 530
+    },
+    {
+      "epoch": 1.7733990147783252,
+      "grad_norm": 0.8341553211212158,
+      "learning_rate": 0.0002,
+      "loss": 1.6945,
+      "step": 540
+    },
+    {
+      "epoch": 1.80623973727422,
+      "grad_norm": 0.6141380071640015,
+      "learning_rate": 0.0002,
+      "loss": 1.6601,
+      "step": 550
+    },
+    {
+      "epoch": 1.839080459770115,
+      "grad_norm": 0.6553818583488464,
+      "learning_rate": 0.0002,
+      "loss": 1.6916,
+      "step": 560
+    },
+    {
+      "epoch": 1.8719211822660098,
+      "grad_norm": 0.675327718257904,
+      "learning_rate": 0.0002,
+      "loss": 1.681,
+      "step": 570
+    },
+    {
+      "epoch": 1.9047619047619047,
+      "grad_norm": 0.6748024821281433,
+      "learning_rate": 0.0002,
+      "loss": 1.6508,
+      "step": 580
+    },
+    {
+      "epoch": 1.9376026272577995,
+      "grad_norm": 0.7145724892616272,
+      "learning_rate": 0.0002,
+      "loss": 1.7898,
+      "step": 590
+    },
+    {
+      "epoch": 1.9704433497536946,
+      "grad_norm": 0.7319425940513611,
+      "learning_rate": 0.0002,
+      "loss": 1.7138,
+      "step": 600
+    },
+    {
+      "epoch": 2.0,
+      "eval_loss": 1.7979766130447388,
+      "eval_runtime": 147.9359,
+      "eval_samples_per_second": 3.603,
+      "eval_steps_per_second": 0.453,
+      "step": 609
+    },
+    {
+      "epoch": 2.0032840722495897,
+      "grad_norm": 0.5940006375312805,
+      "learning_rate": 0.0002,
+      "loss": 1.6249,
+      "step": 610
+    },
+    {
+      "epoch": 2.0361247947454846,
+      "grad_norm": 0.8300226330757141,
+      "learning_rate": 0.0002,
+      "loss": 1.5343,
+      "step": 620
+    },
+    {
+      "epoch": 2.0689655172413794,
+      "grad_norm": 0.7499504089355469,
+      "learning_rate": 0.0002,
+      "loss": 1.5071,
+      "step": 630
+    },
+    {
+      "epoch": 2.1018062397372743,
+      "grad_norm": 0.7384152412414551,
+      "learning_rate": 0.0002,
+      "loss": 1.5293,
+      "step": 640
+    },
+    {
+      "epoch": 2.134646962233169,
+      "grad_norm": 0.7212202548980713,
+      "learning_rate": 0.0002,
+      "loss": 1.4843,
+      "step": 650
+    },
+    {
+      "epoch": 2.167487684729064,
+      "grad_norm": 0.8744934797286987,
+      "learning_rate": 0.0002,
+      "loss": 1.4661,
+      "step": 660
+    },
+    {
+      "epoch": 2.200328407224959,
+      "grad_norm": 0.8432527184486389,
+      "learning_rate": 0.0002,
+      "loss": 1.4922,
+      "step": 670
+    },
+    {
+      "epoch": 2.233169129720854,
+      "grad_norm": 0.8430032730102539,
+      "learning_rate": 0.0002,
+      "loss": 1.4602,
+      "step": 680
+    },
+    {
+      "epoch": 2.2660098522167487,
+      "grad_norm": 0.8931224942207336,
+      "learning_rate": 0.0002,
+      "loss": 1.5109,
+      "step": 690
+    },
+    {
+      "epoch": 2.2988505747126435,
+      "grad_norm": 0.7747994065284729,
+      "learning_rate": 0.0002,
+      "loss": 1.4834,
+      "step": 700
+    },
+    {
+      "epoch": 2.3316912972085384,
+      "grad_norm": 0.839187502861023,
+      "learning_rate": 0.0002,
+      "loss": 1.5328,
+      "step": 710
+    },
+    {
+      "epoch": 2.3645320197044333,
+      "grad_norm": 0.8716842532157898,
+      "learning_rate": 0.0002,
+      "loss": 1.5716,
+      "step": 720
+    },
+    {
+      "epoch": 2.3973727422003286,
+      "grad_norm": 0.9200440049171448,
+      "learning_rate": 0.0002,
+      "loss": 1.5197,
+      "step": 730
+    },
+    {
+      "epoch": 2.4302134646962235,
+      "grad_norm": 0.7892614603042603,
+      "learning_rate": 0.0002,
+      "loss": 1.5512,
+      "step": 740
+    },
+    {
+      "epoch": 2.4630541871921183,
+      "grad_norm": 0.8656499981880188,
+      "learning_rate": 0.0002,
+      "loss": 1.4936,
+      "step": 750
+    },
+    {
+      "epoch": 2.495894909688013,
+      "grad_norm": 1.0396990776062012,
+      "learning_rate": 0.0002,
+      "loss": 1.5198,
+      "step": 760
+    },
+    {
+      "epoch": 2.528735632183908,
+      "grad_norm": 0.8359261751174927,
+      "learning_rate": 0.0002,
+      "loss": 1.5026,
+      "step": 770
+    },
+    {
+      "epoch": 2.561576354679803,
+      "grad_norm": 0.793073296546936,
+      "learning_rate": 0.0002,
+      "loss": 1.5454,
+      "step": 780
+    },
+    {
+      "epoch": 2.594417077175698,
+      "grad_norm": 0.9727579355239868,
+      "learning_rate": 0.0002,
+      "loss": 1.5254,
+      "step": 790
+    },
+    {
+      "epoch": 2.6272577996715927,
+      "grad_norm": 0.8486084342002869,
+      "learning_rate": 0.0002,
+      "loss": 1.5317,
+      "step": 800
+    },
+    {
+      "epoch": 2.6600985221674875,
+      "grad_norm": 0.8329479694366455,
+      "learning_rate": 0.0002,
+      "loss": 1.4904,
+      "step": 810
+    },
+    {
+      "epoch": 2.692939244663383,
+      "grad_norm": 0.8098273873329163,
+      "learning_rate": 0.0002,
+      "loss": 1.5422,
+      "step": 820
+    },
+    {
+      "epoch": 2.7257799671592773,
+      "grad_norm": 0.899185836315155,
+      "learning_rate": 0.0002,
+      "loss": 1.4642,
+      "step": 830
+    },
+    {
+      "epoch": 2.7586206896551726,
+      "grad_norm": 1.03939950466156,
+      "learning_rate": 0.0002,
+      "loss": 1.5208,
+      "step": 840
+    },
+    {
+      "epoch": 2.7914614121510675,
+      "grad_norm": 0.9275932908058167,
+      "learning_rate": 0.0002,
+      "loss": 1.5117,
+      "step": 850
+    },
+    {
+      "epoch": 2.8243021346469623,
+      "grad_norm": 0.8601354956626892,
+      "learning_rate": 0.0002,
+      "loss": 1.5891,
+      "step": 860
+    },
+    {
+      "epoch": 2.857142857142857,
+      "grad_norm": 0.8605263233184814,
+      "learning_rate": 0.0002,
+      "loss": 1.5398,
+      "step": 870
+    },
+    {
+      "epoch": 2.889983579638752,
+      "grad_norm": 2.0008926391601562,
+      "learning_rate": 0.0002,
+      "loss": 1.5964,
+      "step": 880
+    },
+    {
+      "epoch": 2.922824302134647,
+      "grad_norm": 0.8891443610191345,
+      "learning_rate": 0.0002,
+      "loss": 1.5407,
+      "step": 890
+    },
+    {
+      "epoch": 2.955665024630542,
+      "grad_norm": 0.9993823766708374,
+      "learning_rate": 0.0002,
+      "loss": 1.5517,
+      "step": 900
+    },
+    {
+      "epoch": 2.9885057471264367,
+      "grad_norm": 1.07162344455719,
+      "learning_rate": 0.0002,
+      "loss": 1.4897,
+      "step": 910
+    },
+    {
+      "epoch": 2.9983579638752054,
+      "eval_loss": 1.8474907875061035,
+      "eval_runtime": 148.1815,
+      "eval_samples_per_second": 3.597,
+      "eval_steps_per_second": 0.452,
+      "step": 913
+    },
+    {
+      "epoch": 3.0213464696223316,
+      "grad_norm": 1.0293525457382202,
+      "learning_rate": 0.0002,
+      "loss": 1.4149,
+      "step": 920
+    },
+    {
+      "epoch": 3.0541871921182264,
+      "grad_norm": 1.1002044677734375,
+      "learning_rate": 0.0002,
+      "loss": 1.2622,
+      "step": 930
+    },
+    {
+      "epoch": 3.0870279146141213,
+      "grad_norm": 0.9848016500473022,
+      "learning_rate": 0.0002,
+      "loss": 1.23,
+      "step": 940
+    },
+    {
+      "epoch": 3.1198686371100166,
+      "grad_norm": 1.195257306098938,
+      "learning_rate": 0.0002,
+      "loss": 1.2574,
+      "step": 950
+    },
+    {
+      "epoch": 3.1527093596059115,
+      "grad_norm": 1.416608452796936,
+      "learning_rate": 0.0002,
+      "loss": 1.3146,
+      "step": 960
+    },
+    {
+      "epoch": 3.1855500821018063,
+      "grad_norm": 1.1547739505767822,
+      "learning_rate": 0.0002,
+      "loss": 1.2348,
+      "step": 970
+    },
+    {
+      "epoch": 3.218390804597701,
+      "grad_norm": 0.9784232974052429,
+      "learning_rate": 0.0002,
+      "loss": 1.261,
+      "step": 980
+    },
+    {
+      "epoch": 3.251231527093596,
+      "grad_norm": 1.1947113275527954,
+      "learning_rate": 0.0002,
+      "loss": 1.2592,
+      "step": 990
+    },
+    {
+      "epoch": 3.284072249589491,
+      "grad_norm": 1.3581703901290894,
+      "learning_rate": 0.0002,
+      "loss": 1.278,
+      "step": 1000
+    },
+    {
+      "epoch": 3.316912972085386,
+      "grad_norm": 1.1447371244430542,
+      "learning_rate": 0.0002,
+      "loss": 1.2678,
+      "step": 1010
+    },
+    {
+      "epoch": 3.3497536945812807,
+      "grad_norm": 1.3974782228469849,
+      "learning_rate": 0.0002,
+      "loss": 1.3061,
+      "step": 1020
+    },
+    {
+      "epoch": 3.3825944170771756,
+      "grad_norm": 1.5755161046981812,
+      "learning_rate": 0.0002,
+      "loss": 1.2237,
+      "step": 1030
+    },
+    {
+      "epoch": 3.4154351395730704,
+      "grad_norm": 1.2665208578109741,
+      "learning_rate": 0.0002,
+      "loss": 1.2724,
+      "step": 1040
+    },
+    {
+      "epoch": 3.4482758620689653,
+      "grad_norm": 1.1338576078414917,
+      "learning_rate": 0.0002,
+      "loss": 1.3611,
+      "step": 1050
+    },
+    {
+      "epoch": 3.4811165845648606,
+      "grad_norm": 1.179924488067627,
+      "learning_rate": 0.0002,
+      "loss": 1.2958,
+      "step": 1060
+    },
+    {
+      "epoch": 3.5139573070607555,
+      "grad_norm": 1.55759596824646,
+      "learning_rate": 0.0002,
+      "loss": 1.3161,
+      "step": 1070
+    },
+    {
+      "epoch": 3.5467980295566504,
+      "grad_norm": 1.143595576286316,
+      "learning_rate": 0.0002,
+      "loss": 1.3307,
+      "step": 1080
+    },
+    {
+      "epoch": 3.5796387520525452,
+      "grad_norm": 1.1458665132522583,
+      "learning_rate": 0.0002,
+      "loss": 1.3719,
+      "step": 1090
+    },
+    {
+      "epoch": 3.61247947454844,
+      "grad_norm": 1.0880578756332397,
+      "learning_rate": 0.0002,
+      "loss": 1.3359,
+      "step": 1100
+    },
+    {
+      "epoch": 3.645320197044335,
+      "grad_norm": 1.2723114490509033,
+      "learning_rate": 0.0002,
+      "loss": 1.3686,
+      "step": 1110
+    },
+    {
+      "epoch": 3.67816091954023,
+      "grad_norm": 1.9975095987319946,
+      "learning_rate": 0.0002,
+      "loss": 1.3555,
+      "step": 1120
+    },
+    {
+      "epoch": 3.7110016420361247,
+      "grad_norm": 1.3761720657348633,
+      "learning_rate": 0.0002,
+      "loss": 1.2782,
+      "step": 1130
+    },
+    {
+      "epoch": 3.7438423645320196,
+      "grad_norm": 1.2107709646224976,
+      "learning_rate": 0.0002,
+      "loss": 1.3613,
+      "step": 1140
+    },
+    {
+      "epoch": 3.776683087027915,
+      "grad_norm": 1.16817307472229,
+      "learning_rate": 0.0002,
+      "loss": 1.3707,
+      "step": 1150
+    },
+    {
+      "epoch": 3.8095238095238093,
+      "grad_norm": 1.2264677286148071,
+      "learning_rate": 0.0002,
+      "loss": 1.3055,
+      "step": 1160
+    },
+    {
+      "epoch": 3.8423645320197046,
+      "grad_norm": 1.2869375944137573,
+      "learning_rate": 0.0002,
+      "loss": 1.3441,
+      "step": 1170
+    },
+    {
+      "epoch": 3.8752052545155995,
+      "grad_norm": 1.8852620124816895,
+      "learning_rate": 0.0002,
+      "loss": 1.3593,
+      "step": 1180
+    },
+    {
+      "epoch": 3.9080459770114944,
+      "grad_norm": 1.0901968479156494,
+      "learning_rate": 0.0002,
+      "loss": 1.3691,
+      "step": 1190
+    },
+    {
+      "epoch": 3.9408866995073892,
+      "grad_norm": 1.130061388015747,
+      "learning_rate": 0.0002,
+      "loss": 1.2799,
+      "step": 1200
+    },
+    {
+      "epoch": 3.973727422003284,
+      "grad_norm": 1.267082929611206,
+      "learning_rate": 0.0002,
+      "loss": 1.3157,
+      "step": 1210
+    },
+    {
+      "epoch": 4.0,
+      "eval_loss": 1.9435356855392456,
+      "eval_runtime": 152.8355,
+      "eval_samples_per_second": 3.487,
+      "eval_steps_per_second": 0.438,
+      "step": 1218
+    }
+  ],
+  "logging_steps": 10,
+  "max_steps": 2432,
+  "num_input_tokens_seen": 0,
+  "num_train_epochs": 8,
+  "save_steps": 200,
+  "stateful_callbacks": {
+    "TrainerControl": {
+      "args": {
+        "should_epoch_stop": false,
+        "should_evaluate": false,
+        "should_log": false,
+        "should_save": true,
+        "should_training_stop": false
+      },
+      "attributes": {}
+    }
+  },
+  "total_flos": 5.34394171293696e+16,
+  "train_batch_size": 1,
+  "trial_name": null,
+  "trial_params": null
+}

Mistral-7B-Instruct-v0.3_int4_hellaswag-routerbench-0shot_lr-0.0002_e-8_seq-512_lora-a-32-d-0.05-r-64_bs-1_gas-2_tf32-True_tunedata-portion-p-0.2-num-1405-sd-4/checkpoint-1218/training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:726dbd1b57eddad44e9f6bd0029e07b38acb3489720ae7c0f0fc0c9df3587bab
+size 5560

Mistral-7B-Instruct-v0.3_int4_hellaswag-routerbench-0shot_lr-0.0002_e-8_seq-512_lora-a-32-d-0.05-r-64_bs-1_gas-2_tf32-True_tunedata-portion-p-0.2-num-1405-sd-4/checkpoint-1522/README.md

@@ -0,0 +1,202 @@
+---
+base_model: mistralai/Mistral-7B-Instruct-v0.3
+library_name: peft
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.13.1

Mistral-7B-Instruct-v0.3_int4_hellaswag-routerbench-0shot_lr-0.0002_e-8_seq-512_lora-a-32-d-0.05-r-64_bs-1_gas-2_tf32-True_tunedata-portion-p-0.2-num-1405-sd-4/checkpoint-1522/adapter_config.json

@@ -0,0 +1,29 @@
+{
+  "alpha_pattern": {},
+  "auto_mapping": null,
+  "base_model_name_or_path": "mistralai/Mistral-7B-Instruct-v0.3",
+  "bias": "none",
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 32,
+  "lora_dropout": 0.05,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "r": 64,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "q_proj",
+    "v_proj"
+  ],
+  "task_type": "CAUSAL_LM",
+  "use_dora": false,
+  "use_rslora": false
+}

Mistral-7B-Instruct-v0.3_int4_hellaswag-routerbench-0shot_lr-0.0002_e-8_seq-512_lora-a-32-d-0.05-r-64_bs-1_gas-2_tf32-True_tunedata-portion-p-0.2-num-1405-sd-4/checkpoint-1522/adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:dbdb608e848f83c67df394382b491f2fdc026a36895e09c9b9f7cd2d4a958b1b
+size 109069176

Mistral-7B-Instruct-v0.3_int4_hellaswag-routerbench-0shot_lr-0.0002_e-8_seq-512_lora-a-32-d-0.05-r-64_bs-1_gas-2_tf32-True_tunedata-portion-p-0.2-num-1405-sd-4/checkpoint-1522/optimizer.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3aa02b357c27ee0e74bc4c997b7bf51ee6331d318e64a07c0984e236849d7ccf
+size 55532666

Mistral-7B-Instruct-v0.3_int4_hellaswag-routerbench-0shot_lr-0.0002_e-8_seq-512_lora-a-32-d-0.05-r-64_bs-1_gas-2_tf32-True_tunedata-portion-p-0.2-num-1405-sd-4/checkpoint-1522/rng_state.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:61b0b200d3126ae5e151414061e2bb0b03de70ebf3e8b8e20658db22544aa9c8
+size 14244

Mistral-7B-Instruct-v0.3_int4_hellaswag-routerbench-0shot_lr-0.0002_e-8_seq-512_lora-a-32-d-0.05-r-64_bs-1_gas-2_tf32-True_tunedata-portion-p-0.2-num-1405-sd-4/checkpoint-1522/scheduler.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f714bc1f06f7de36e56d219bd9373498f7baac357a6ba583bd3c8ca86d1e072a
+size 1064

Mistral-7B-Instruct-v0.3_int4_hellaswag-routerbench-0shot_lr-0.0002_e-8_seq-512_lora-a-32-d-0.05-r-64_bs-1_gas-2_tf32-True_tunedata-portion-p-0.2-num-1405-sd-4/checkpoint-1522/special_tokens_map.json

@@ -0,0 +1,24 @@
+{
+  "bos_token": {
+    "content": "<s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "</s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": "</s>",
+  "unk_token": {
+    "content": "<unk>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

Mistral-7B-Instruct-v0.3_int4_hellaswag-routerbench-0shot_lr-0.0002_e-8_seq-512_lora-a-32-d-0.05-r-64_bs-1_gas-2_tf32-True_tunedata-portion-p-0.2-num-1405-sd-4/checkpoint-1522/tokenizer.model

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:37f00374dea48658ee8f5d0f21895b9bc55cb0103939607c8185bfd1c6ca1f89
+size 587404

Mistral-7B-Instruct-v0.3_int4_hellaswag-routerbench-0shot_lr-0.0002_e-8_seq-512_lora-a-32-d-0.05-r-64_bs-1_gas-2_tf32-True_tunedata-portion-p-0.2-num-1405-sd-4/checkpoint-1522/trainer_state.json

@@ -0,0 +1,1137 @@
+{
+  "best_metric": 1.790574550628662,
+  "best_model_checkpoint": "outputs-001/Mistral-7B-Instruct-v0.3_int4_hellaswag-routerbench-0shot_lr-0.0002_e-8_seq-512_lora-a-32-d-0.05-r-64_bs-1_gas-2_tf32-True_tunedata-portion-p-0.2-num-1405-sd-4/checkpoint-304",
+  "epoch": 4.998357963875205,
+  "eval_steps": 10,
+  "global_step": 1522,
+  "is_hyper_param_search": false,
+  "is_local_process_zero": true,
+  "is_world_process_zero": true,
+  "log_history": [
+    {
+      "epoch": 0.03284072249589491,
+      "grad_norm": 0.8926695585250854,
+      "learning_rate": 0.0002,
+      "loss": 2.7246,
+      "step": 10
+    },
+    {
+      "epoch": 0.06568144499178982,
+      "grad_norm": 1.181358814239502,
+      "learning_rate": 0.0002,
+      "loss": 2.1913,
+      "step": 20
+    },
+    {
+      "epoch": 0.09852216748768473,
+      "grad_norm": 0.7392292618751526,
+      "learning_rate": 0.0002,
+      "loss": 2.1022,
+      "step": 30
+    },
+    {
+      "epoch": 0.13136288998357964,
+      "grad_norm": 0.7944067716598511,
+      "learning_rate": 0.0002,
+      "loss": 2.071,
+      "step": 40
+    },
+    {
+      "epoch": 0.16420361247947454,
+      "grad_norm": 0.7858729958534241,
+      "learning_rate": 0.0002,
+      "loss": 2.0564,
+      "step": 50
+    },
+    {
+      "epoch": 0.19704433497536947,
+      "grad_norm": 0.7574328184127808,
+      "learning_rate": 0.0002,
+      "loss": 2.065,
+      "step": 60
+    },
+    {
+      "epoch": 0.22988505747126436,
+      "grad_norm": 0.694824755191803,
+      "learning_rate": 0.0002,
+      "loss": 1.9014,
+      "step": 70
+    },
+    {
+      "epoch": 0.2627257799671593,
+      "grad_norm": 0.6516484022140503,
+      "learning_rate": 0.0002,
+      "loss": 1.885,
+      "step": 80
+    },
+    {
+      "epoch": 0.2955665024630542,
+      "grad_norm": 0.6870642304420471,
+      "learning_rate": 0.0002,
+      "loss": 1.8555,
+      "step": 90
+    },
+    {
+      "epoch": 0.3284072249589491,
+      "grad_norm": 0.7598554491996765,
+      "learning_rate": 0.0002,
+      "loss": 1.84,
+      "step": 100
+    },
+    {
+      "epoch": 0.361247947454844,
+      "grad_norm": 0.6050140857696533,
+      "learning_rate": 0.0002,
+      "loss": 1.8204,
+      "step": 110
+    },
+    {
+      "epoch": 0.39408866995073893,
+      "grad_norm": 0.7373790740966797,
+      "learning_rate": 0.0002,
+      "loss": 1.8294,
+      "step": 120
+    },
+    {
+      "epoch": 0.4269293924466338,
+      "grad_norm": 0.6588838696479797,
+      "learning_rate": 0.0002,
+      "loss": 1.7682,
+      "step": 130
+    },
+    {
+      "epoch": 0.45977011494252873,
+      "grad_norm": 0.5314497947692871,
+      "learning_rate": 0.0002,
+      "loss": 1.7925,
+      "step": 140
+    },
+    {
+      "epoch": 0.49261083743842365,
+      "grad_norm": 0.6273255944252014,
+      "learning_rate": 0.0002,
+      "loss": 1.7992,
+      "step": 150
+    },
+    {
+      "epoch": 0.5254515599343186,
+      "grad_norm": 0.5744606256484985,
+      "learning_rate": 0.0002,
+      "loss": 1.8207,
+      "step": 160
+    },
+    {
+      "epoch": 0.5582922824302134,
+      "grad_norm": 0.577231228351593,
+      "learning_rate": 0.0002,
+      "loss": 1.8215,
+      "step": 170
+    },
+    {
+      "epoch": 0.5911330049261084,
+      "grad_norm": 0.5391182899475098,
+      "learning_rate": 0.0002,
+      "loss": 1.8125,
+      "step": 180
+    },
+    {
+      "epoch": 0.6239737274220033,
+      "grad_norm": 0.5863330364227295,
+      "learning_rate": 0.0002,
+      "loss": 1.8501,
+      "step": 190
+    },
+    {
+      "epoch": 0.6568144499178982,
+      "grad_norm": 0.6266984343528748,
+      "learning_rate": 0.0002,
+      "loss": 1.8055,
+      "step": 200
+    },
+    {
+      "epoch": 0.6896551724137931,
+      "grad_norm": 0.5842767357826233,
+      "learning_rate": 0.0002,
+      "loss": 1.7923,
+      "step": 210
+    },
+    {
+      "epoch": 0.722495894909688,
+      "grad_norm": 0.5255483984947205,
+      "learning_rate": 0.0002,
+      "loss": 1.7906,
+      "step": 220
+    },
+    {
+      "epoch": 0.7553366174055829,
+      "grad_norm": 0.7259632349014282,
+      "learning_rate": 0.0002,
+      "loss": 1.786,
+      "step": 230
+    },
+    {
+      "epoch": 0.7881773399014779,
+      "grad_norm": 0.5593494176864624,
+      "learning_rate": 0.0002,
+      "loss": 1.7291,
+      "step": 240
+    },
+    {
+      "epoch": 0.8210180623973727,
+      "grad_norm": 0.5614203214645386,
+      "learning_rate": 0.0002,
+      "loss": 1.8098,
+      "step": 250
+    },
+    {
+      "epoch": 0.8538587848932676,
+      "grad_norm": 0.6784001588821411,
+      "learning_rate": 0.0002,
+      "loss": 1.814,
+      "step": 260
+    },
+    {
+      "epoch": 0.8866995073891626,
+      "grad_norm": 0.5510041117668152,
+      "learning_rate": 0.0002,
+      "loss": 1.6856,
+      "step": 270
+    },
+    {
+      "epoch": 0.9195402298850575,
+      "grad_norm": 0.5453769564628601,
+      "learning_rate": 0.0002,
+      "loss": 1.7939,
+      "step": 280
+    },
+    {
+      "epoch": 0.9523809523809523,
+      "grad_norm": 0.5494107604026794,
+      "learning_rate": 0.0002,
+      "loss": 1.7489,
+      "step": 290
+    },
+    {
+      "epoch": 0.9852216748768473,
+      "grad_norm": 0.5768024921417236,
+      "learning_rate": 0.0002,
+      "loss": 1.8442,
+      "step": 300
+    },
+    {
+      "epoch": 0.9983579638752053,
+      "eval_loss": 1.790574550628662,
+      "eval_runtime": 147.7829,
+      "eval_samples_per_second": 3.607,
+      "eval_steps_per_second": 0.453,
+      "step": 304
+    },
+    {
+      "epoch": 1.0180623973727423,
+      "grad_norm": 0.5931118130683899,
+      "learning_rate": 0.0002,
+      "loss": 1.7061,
+      "step": 310
+    },
+    {
+      "epoch": 1.0509031198686372,
+      "grad_norm": 0.6433814764022827,
+      "learning_rate": 0.0002,
+      "loss": 1.6997,
+      "step": 320
+    },
+    {
+      "epoch": 1.083743842364532,
+      "grad_norm": 0.6801510453224182,
+      "learning_rate": 0.0002,
+      "loss": 1.697,
+      "step": 330
+    },
+    {
+      "epoch": 1.116584564860427,
+      "grad_norm": 0.6511064171791077,
+      "learning_rate": 0.0002,
+      "loss": 1.721,
+      "step": 340
+    },
+    {
+      "epoch": 1.1494252873563218,
+      "grad_norm": 0.6265677213668823,
+      "learning_rate": 0.0002,
+      "loss": 1.7209,
+      "step": 350
+    },
+    {
+      "epoch": 1.1822660098522166,
+      "grad_norm": 0.6211821436882019,
+      "learning_rate": 0.0002,
+      "loss": 1.6524,
+      "step": 360
+    },
+    {
+      "epoch": 1.2151067323481117,
+      "grad_norm": 0.6931810975074768,
+      "learning_rate": 0.0002,
+      "loss": 1.7091,
+      "step": 370
+    },
+    {
+      "epoch": 1.2479474548440066,
+      "grad_norm": 0.5978460311889648,
+      "learning_rate": 0.0002,
+      "loss": 1.6604,
+      "step": 380
+    },
+    {
+      "epoch": 1.2807881773399015,
+      "grad_norm": 0.6581379175186157,
+      "learning_rate": 0.0002,
+      "loss": 1.6707,
+      "step": 390
+    },
+    {
+      "epoch": 1.3136288998357963,
+      "grad_norm": 0.6794891953468323,
+      "learning_rate": 0.0002,
+      "loss": 1.6543,
+      "step": 400
+    },
+    {
+      "epoch": 1.3464696223316914,
+      "grad_norm": 0.6028420329093933,
+      "learning_rate": 0.0002,
+      "loss": 1.7261,
+      "step": 410
+    },
+    {
+      "epoch": 1.3793103448275863,
+      "grad_norm": 0.8568810820579529,
+      "learning_rate": 0.0002,
+      "loss": 1.7234,
+      "step": 420
+    },
+    {
+      "epoch": 1.4121510673234812,
+      "grad_norm": 0.6654542088508606,
+      "learning_rate": 0.0002,
+      "loss": 1.6885,
+      "step": 430
+    },
+    {
+      "epoch": 1.444991789819376,
+      "grad_norm": 0.6363505721092224,
+      "learning_rate": 0.0002,
+      "loss": 1.7335,
+      "step": 440
+    },
+    {
+      "epoch": 1.477832512315271,
+      "grad_norm": 0.7018159627914429,
+      "learning_rate": 0.0002,
+      "loss": 1.6947,
+      "step": 450
+    },
+    {
+      "epoch": 1.5106732348111658,
+      "grad_norm": 0.6718074679374695,
+      "learning_rate": 0.0002,
+      "loss": 1.7006,
+      "step": 460
+    },
+    {
+      "epoch": 1.5435139573070606,
+      "grad_norm": 0.6456331610679626,
+      "learning_rate": 0.0002,
+      "loss": 1.6504,
+      "step": 470
+    },
+    {
+      "epoch": 1.5763546798029555,
+      "grad_norm": 0.7088801860809326,
+      "learning_rate": 0.0002,
+      "loss": 1.6953,
+      "step": 480
+    },
+    {
+      "epoch": 1.6091954022988506,
+      "grad_norm": 0.8001429438591003,
+      "learning_rate": 0.0002,
+      "loss": 1.6521,
+      "step": 490
+    },
+    {
+      "epoch": 1.6420361247947455,
+      "grad_norm": 0.6954358816146851,
+      "learning_rate": 0.0002,
+      "loss": 1.6384,
+      "step": 500
+    },
+    {
+      "epoch": 1.6748768472906403,
+      "grad_norm": 0.6385519504547119,
+      "learning_rate": 0.0002,
+      "loss": 1.6606,
+      "step": 510
+    },
+    {
+      "epoch": 1.7077175697865354,
+      "grad_norm": 0.6736445426940918,
+      "learning_rate": 0.0002,
+      "loss": 1.7151,
+      "step": 520
+    },
+    {
+      "epoch": 1.7405582922824303,
+      "grad_norm": 0.6794690489768982,
+      "learning_rate": 0.0002,
+      "loss": 1.7363,
+      "step": 530
+    },
+    {
+      "epoch": 1.7733990147783252,
+      "grad_norm": 0.8341553211212158,
+      "learning_rate": 0.0002,
+      "loss": 1.6945,
+      "step": 540
+    },
+    {
+      "epoch": 1.80623973727422,
+      "grad_norm": 0.6141380071640015,
+      "learning_rate": 0.0002,
+      "loss": 1.6601,
+      "step": 550
+    },
+    {
+      "epoch": 1.839080459770115,
+      "grad_norm": 0.6553818583488464,
+      "learning_rate": 0.0002,
+      "loss": 1.6916,
+      "step": 560
+    },
+    {
+      "epoch": 1.8719211822660098,
+      "grad_norm": 0.675327718257904,
+      "learning_rate": 0.0002,
+      "loss": 1.681,
+      "step": 570
+    },
+    {
+      "epoch": 1.9047619047619047,
+      "grad_norm": 0.6748024821281433,
+      "learning_rate": 0.0002,
+      "loss": 1.6508,
+      "step": 580
+    },
+    {
+      "epoch": 1.9376026272577995,
+      "grad_norm": 0.7145724892616272,
+      "learning_rate": 0.0002,
+      "loss": 1.7898,
+      "step": 590
+    },
+    {
+      "epoch": 1.9704433497536946,
+      "grad_norm": 0.7319425940513611,
+      "learning_rate": 0.0002,
+      "loss": 1.7138,
+      "step": 600
+    },
+    {
+      "epoch": 2.0,
+      "eval_loss": 1.7979766130447388,
+      "eval_runtime": 147.9359,
+      "eval_samples_per_second": 3.603,
+      "eval_steps_per_second": 0.453,
+      "step": 609
+    },
+    {
+      "epoch": 2.0032840722495897,
+      "grad_norm": 0.5940006375312805,
+      "learning_rate": 0.0002,
+      "loss": 1.6249,
+      "step": 610
+    },
+    {
+      "epoch": 2.0361247947454846,
+      "grad_norm": 0.8300226330757141,
+      "learning_rate": 0.0002,
+      "loss": 1.5343,
+      "step": 620
+    },
+    {
+      "epoch": 2.0689655172413794,
+      "grad_norm": 0.7499504089355469,
+      "learning_rate": 0.0002,
+      "loss": 1.5071,
+      "step": 630
+    },
+    {
+      "epoch": 2.1018062397372743,
+      "grad_norm": 0.7384152412414551,
+      "learning_rate": 0.0002,
+      "loss": 1.5293,
+      "step": 640
+    },
+    {
+      "epoch": 2.134646962233169,
+      "grad_norm": 0.7212202548980713,
+      "learning_rate": 0.0002,
+      "loss": 1.4843,
+      "step": 650
+    },
+    {
+      "epoch": 2.167487684729064,
+      "grad_norm": 0.8744934797286987,
+      "learning_rate": 0.0002,
+      "loss": 1.4661,
+      "step": 660
+    },
+    {
+      "epoch": 2.200328407224959,
+      "grad_norm": 0.8432527184486389,
+      "learning_rate": 0.0002,
+      "loss": 1.4922,
+      "step": 670
+    },
+    {
+      "epoch": 2.233169129720854,
+      "grad_norm": 0.8430032730102539,
+      "learning_rate": 0.0002,
+      "loss": 1.4602,
+      "step": 680
+    },
+    {
+      "epoch": 2.2660098522167487,
+      "grad_norm": 0.8931224942207336,
+      "learning_rate": 0.0002,
+      "loss": 1.5109,
+      "step": 690
+    },
+    {
+      "epoch": 2.2988505747126435,
+      "grad_norm": 0.7747994065284729,
+      "learning_rate": 0.0002,
+      "loss": 1.4834,
+      "step": 700
+    },
+    {
+      "epoch": 2.3316912972085384,
+      "grad_norm": 0.839187502861023,
+      "learning_rate": 0.0002,
+      "loss": 1.5328,
+      "step": 710
+    },
+    {
+      "epoch": 2.3645320197044333,
+      "grad_norm": 0.8716842532157898,
+      "learning_rate": 0.0002,
+      "loss": 1.5716,
+      "step": 720
+    },
+    {
+      "epoch": 2.3973727422003286,
+      "grad_norm": 0.9200440049171448,
+      "learning_rate": 0.0002,
+      "loss": 1.5197,
+      "step": 730
+    },
+    {
+      "epoch": 2.4302134646962235,
+      "grad_norm": 0.7892614603042603,
+      "learning_rate": 0.0002,
+      "loss": 1.5512,
+      "step": 740
+    },
+    {
+      "epoch": 2.4630541871921183,
+      "grad_norm": 0.8656499981880188,
+      "learning_rate": 0.0002,
+      "loss": 1.4936,
+      "step": 750
+    },
+    {
+      "epoch": 2.495894909688013,
+      "grad_norm": 1.0396990776062012,
+      "learning_rate": 0.0002,
+      "loss": 1.5198,
+      "step": 760
+    },
+    {
+      "epoch": 2.528735632183908,
+      "grad_norm": 0.8359261751174927,
+      "learning_rate": 0.0002,
+      "loss": 1.5026,
+      "step": 770
+    },
+    {
+      "epoch": 2.561576354679803,
+      "grad_norm": 0.793073296546936,
+      "learning_rate": 0.0002,
+      "loss": 1.5454,
+      "step": 780
+    },
+    {
+      "epoch": 2.594417077175698,
+      "grad_norm": 0.9727579355239868,
+      "learning_rate": 0.0002,
+      "loss": 1.5254,
+      "step": 790
+    },
+    {
+      "epoch": 2.6272577996715927,
+      "grad_norm": 0.8486084342002869,
+      "learning_rate": 0.0002,
+      "loss": 1.5317,
+      "step": 800
+    },
+    {
+      "epoch": 2.6600985221674875,
+      "grad_norm": 0.8329479694366455,
+      "learning_rate": 0.0002,
+      "loss": 1.4904,
+      "step": 810
+    },
+    {
+      "epoch": 2.692939244663383,
+      "grad_norm": 0.8098273873329163,
+      "learning_rate": 0.0002,
+      "loss": 1.5422,
+      "step": 820
+    },
+    {
+      "epoch": 2.7257799671592773,
+      "grad_norm": 0.899185836315155,
+      "learning_rate": 0.0002,
+      "loss": 1.4642,
+      "step": 830
+    },
+    {
+      "epoch": 2.7586206896551726,
+      "grad_norm": 1.03939950466156,
+      "learning_rate": 0.0002,
+      "loss": 1.5208,
+      "step": 840
+    },
+    {
+      "epoch": 2.7914614121510675,
+      "grad_norm": 0.9275932908058167,
+      "learning_rate": 0.0002,
+      "loss": 1.5117,
+      "step": 850
+    },
+    {
+      "epoch": 2.8243021346469623,
+      "grad_norm": 0.8601354956626892,
+      "learning_rate": 0.0002,
+      "loss": 1.5891,
+      "step": 860
+    },
+    {
+      "epoch": 2.857142857142857,
+      "grad_norm": 0.8605263233184814,
+      "learning_rate": 0.0002,
+      "loss": 1.5398,
+      "step": 870
+    },
+    {
+      "epoch": 2.889983579638752,
+      "grad_norm": 2.0008926391601562,
+      "learning_rate": 0.0002,
+      "loss": 1.5964,
+      "step": 880
+    },
+    {
+      "epoch": 2.922824302134647,
+      "grad_norm": 0.8891443610191345,
+      "learning_rate": 0.0002,
+      "loss": 1.5407,
+      "step": 890
+    },
+    {
+      "epoch": 2.955665024630542,
+      "grad_norm": 0.9993823766708374,
+      "learning_rate": 0.0002,
+      "loss": 1.5517,
+      "step": 900
+    },
+    {
+      "epoch": 2.9885057471264367,
+      "grad_norm": 1.07162344455719,
+      "learning_rate": 0.0002,
+      "loss": 1.4897,
+      "step": 910
+    },
+    {
+      "epoch": 2.9983579638752054,
+      "eval_loss": 1.8474907875061035,
+      "eval_runtime": 148.1815,
+      "eval_samples_per_second": 3.597,
+      "eval_steps_per_second": 0.452,
+      "step": 913
+    },
+    {
+      "epoch": 3.0213464696223316,
+      "grad_norm": 1.0293525457382202,
+      "learning_rate": 0.0002,
+      "loss": 1.4149,
+      "step": 920
+    },
+    {
+      "epoch": 3.0541871921182264,
+      "grad_norm": 1.1002044677734375,
+      "learning_rate": 0.0002,
+      "loss": 1.2622,
+      "step": 930
+    },
+    {
+      "epoch": 3.0870279146141213,
+      "grad_norm": 0.9848016500473022,
+      "learning_rate": 0.0002,
+      "loss": 1.23,
+      "step": 940
+    },
+    {
+      "epoch": 3.1198686371100166,
+      "grad_norm": 1.195257306098938,
+      "learning_rate": 0.0002,
+      "loss": 1.2574,
+      "step": 950
+    },
+    {
+      "epoch": 3.1527093596059115,
+      "grad_norm": 1.416608452796936,
+      "learning_rate": 0.0002,
+      "loss": 1.3146,
+      "step": 960
+    },
+    {
+      "epoch": 3.1855500821018063,
+      "grad_norm": 1.1547739505767822,
+      "learning_rate": 0.0002,
+      "loss": 1.2348,
+      "step": 970
+    },
+    {
+      "epoch": 3.218390804597701,
+      "grad_norm": 0.9784232974052429,
+      "learning_rate": 0.0002,
+      "loss": 1.261,
+      "step": 980
+    },
+    {
+      "epoch": 3.251231527093596,
+      "grad_norm": 1.1947113275527954,
+      "learning_rate": 0.0002,
+      "loss": 1.2592,
+      "step": 990
+    },
+    {
+      "epoch": 3.284072249589491,
+      "grad_norm": 1.3581703901290894,
+      "learning_rate": 0.0002,
+      "loss": 1.278,
+      "step": 1000
+    },
+    {
+      "epoch": 3.316912972085386,
+      "grad_norm": 1.1447371244430542,
+      "learning_rate": 0.0002,
+      "loss": 1.2678,
+      "step": 1010
+    },
+    {
+      "epoch": 3.3497536945812807,
+      "grad_norm": 1.3974782228469849,
+      "learning_rate": 0.0002,
+      "loss": 1.3061,
+      "step": 1020
+    },
+    {
+      "epoch": 3.3825944170771756,
+      "grad_norm": 1.5755161046981812,
+      "learning_rate": 0.0002,
+      "loss": 1.2237,
+      "step": 1030
+    },
+    {
+      "epoch": 3.4154351395730704,
+      "grad_norm": 1.2665208578109741,
+      "learning_rate": 0.0002,
+      "loss": 1.2724,
+      "step": 1040
+    },
+    {
+      "epoch": 3.4482758620689653,
+      "grad_norm": 1.1338576078414917,
+      "learning_rate": 0.0002,
+      "loss": 1.3611,
+      "step": 1050
+    },
+    {
+      "epoch": 3.4811165845648606,
+      "grad_norm": 1.179924488067627,
+      "learning_rate": 0.0002,
+      "loss": 1.2958,
+      "step": 1060
+    },
+    {
+      "epoch": 3.5139573070607555,
+      "grad_norm": 1.55759596824646,
+      "learning_rate": 0.0002,
+      "loss": 1.3161,
+      "step": 1070
+    },
+    {
+      "epoch": 3.5467980295566504,
+      "grad_norm": 1.143595576286316,
+      "learning_rate": 0.0002,
+      "loss": 1.3307,
+      "step": 1080
+    },
+    {
+      "epoch": 3.5796387520525452,
+      "grad_norm": 1.1458665132522583,
+      "learning_rate": 0.0002,
+      "loss": 1.3719,
+      "step": 1090
+    },
+    {
+      "epoch": 3.61247947454844,
+      "grad_norm": 1.0880578756332397,
+      "learning_rate": 0.0002,
+      "loss": 1.3359,
+      "step": 1100
+    },
+    {
+      "epoch": 3.645320197044335,
+      "grad_norm": 1.2723114490509033,
+      "learning_rate": 0.0002,
+      "loss": 1.3686,
+      "step": 1110
+    },
+    {
+      "epoch": 3.67816091954023,
+      "grad_norm": 1.9975095987319946,
+      "learning_rate": 0.0002,
+      "loss": 1.3555,
+      "step": 1120
+    },
+    {
+      "epoch": 3.7110016420361247,
+      "grad_norm": 1.3761720657348633,
+      "learning_rate": 0.0002,
+      "loss": 1.2782,
+      "step": 1130
+    },
+    {
+      "epoch": 3.7438423645320196,
+      "grad_norm": 1.2107709646224976,
+      "learning_rate": 0.0002,
+      "loss": 1.3613,
+      "step": 1140
+    },
+    {
+      "epoch": 3.776683087027915,
+      "grad_norm": 1.16817307472229,
+      "learning_rate": 0.0002,
+      "loss": 1.3707,
+      "step": 1150
+    },
+    {
+      "epoch": 3.8095238095238093,
+      "grad_norm": 1.2264677286148071,
+      "learning_rate": 0.0002,
+      "loss": 1.3055,
+      "step": 1160
+    },
+    {
+      "epoch": 3.8423645320197046,
+      "grad_norm": 1.2869375944137573,
+      "learning_rate": 0.0002,
+      "loss": 1.3441,
+      "step": 1170
+    },
+    {
+      "epoch": 3.8752052545155995,
+      "grad_norm": 1.8852620124816895,
+      "learning_rate": 0.0002,
+      "loss": 1.3593,
+      "step": 1180
+    },
+    {
+      "epoch": 3.9080459770114944,
+      "grad_norm": 1.0901968479156494,
+      "learning_rate": 0.0002,
+      "loss": 1.3691,
+      "step": 1190
+    },
+    {
+      "epoch": 3.9408866995073892,
+      "grad_norm": 1.130061388015747,
+      "learning_rate": 0.0002,
+      "loss": 1.2799,
+      "step": 1200
+    },
+    {
+      "epoch": 3.973727422003284,
+      "grad_norm": 1.267082929611206,
+      "learning_rate": 0.0002,
+      "loss": 1.3157,
+      "step": 1210
+    },
+    {
+      "epoch": 4.0,
+      "eval_loss": 1.9435356855392456,
+      "eval_runtime": 152.8355,
+      "eval_samples_per_second": 3.487,
+      "eval_steps_per_second": 0.438,
+      "step": 1218
+    },
+    {
+      "epoch": 4.006568144499179,
+      "grad_norm": 1.1177356243133545,
+      "learning_rate": 0.0002,
+      "loss": 1.2551,
+      "step": 1220
+    },
+    {
+      "epoch": 4.039408866995074,
+      "grad_norm": 1.30461585521698,
+      "learning_rate": 0.0002,
+      "loss": 1.0096,
+      "step": 1230
+    },
+    {
+      "epoch": 4.072249589490969,
+      "grad_norm": 1.863768219947815,
+      "learning_rate": 0.0002,
+      "loss": 0.9757,
+      "step": 1240
+    },
+    {
+      "epoch": 4.105090311986864,
+      "grad_norm": 1.4688764810562134,
+      "learning_rate": 0.0002,
+      "loss": 1.1466,
+      "step": 1250
+    },
+    {
+      "epoch": 4.137931034482759,
+      "grad_norm": 1.3299670219421387,
+      "learning_rate": 0.0002,
+      "loss": 1.1102,
+      "step": 1260
+    },
+    {
+      "epoch": 4.170771756978653,
+      "grad_norm": 1.726919412612915,
+      "learning_rate": 0.0002,
+      "loss": 0.9916,
+      "step": 1270
+    },
+    {
+      "epoch": 4.203612479474549,
+      "grad_norm": 1.6402113437652588,
+      "learning_rate": 0.0002,
+      "loss": 1.137,
+      "step": 1280
+    },
+    {
+      "epoch": 4.236453201970443,
+      "grad_norm": 1.6000759601593018,
+      "learning_rate": 0.0002,
+      "loss": 1.0551,
+      "step": 1290
+    },
+    {
+      "epoch": 4.269293924466338,
+      "grad_norm": 1.4904356002807617,
+      "learning_rate": 0.0002,
+      "loss": 1.1378,
+      "step": 1300
+    },
+    {
+      "epoch": 4.302134646962233,
+      "grad_norm": 1.8119593858718872,
+      "learning_rate": 0.0002,
+      "loss": 1.0761,
+      "step": 1310
+    },
+    {
+      "epoch": 4.334975369458128,
+      "grad_norm": 1.5564337968826294,
+      "learning_rate": 0.0002,
+      "loss": 1.0773,
+      "step": 1320
+    },
+    {
+      "epoch": 4.3678160919540225,
+      "grad_norm": 1.6291178464889526,
+      "learning_rate": 0.0002,
+      "loss": 1.0441,
+      "step": 1330
+    },
+    {
+      "epoch": 4.400656814449918,
+      "grad_norm": 1.691028356552124,
+      "learning_rate": 0.0002,
+      "loss": 1.0053,
+      "step": 1340
+    },
+    {
+      "epoch": 4.433497536945813,
+      "grad_norm": 1.7010000944137573,
+      "learning_rate": 0.0002,
+      "loss": 1.035,
+      "step": 1350
+    },
+    {
+      "epoch": 4.466338259441708,
+      "grad_norm": 2.038245439529419,
+      "learning_rate": 0.0002,
+      "loss": 1.0319,
+      "step": 1360
+    },
+    {
+      "epoch": 4.499178981937603,
+      "grad_norm": 1.5663248300552368,
+      "learning_rate": 0.0002,
+      "loss": 1.0681,
+      "step": 1370
+    },
+    {
+      "epoch": 4.532019704433497,
+      "grad_norm": 1.6198927164077759,
+      "learning_rate": 0.0002,
+      "loss": 1.1214,
+      "step": 1380
+    },
+    {
+      "epoch": 4.564860426929393,
+      "grad_norm": 1.53022301197052,
+      "learning_rate": 0.0002,
+      "loss": 1.0945,
+      "step": 1390
+    },
+    {
+      "epoch": 4.597701149425287,
+      "grad_norm": 1.4284729957580566,
+      "learning_rate": 0.0002,
+      "loss": 1.1352,
+      "step": 1400
+    },
+    {
+      "epoch": 4.630541871921182,
+      "grad_norm": 1.741274118423462,
+      "learning_rate": 0.0002,
+      "loss": 1.0917,
+      "step": 1410
+    },
+    {
+      "epoch": 4.663382594417077,
+      "grad_norm": 1.5251778364181519,
+      "learning_rate": 0.0002,
+      "loss": 1.081,
+      "step": 1420
+    },
+    {
+      "epoch": 4.696223316912972,
+      "grad_norm": 1.9084422588348389,
+      "learning_rate": 0.0002,
+      "loss": 1.0455,
+      "step": 1430
+    },
+    {
+      "epoch": 4.7290640394088665,
+      "grad_norm": 1.868015170097351,
+      "learning_rate": 0.0002,
+      "loss": 1.0751,
+      "step": 1440
+    },
+    {
+      "epoch": 4.761904761904762,
+      "grad_norm": 1.634352445602417,
+      "learning_rate": 0.0002,
+      "loss": 1.1199,
+      "step": 1450
+    },
+    {
+      "epoch": 4.794745484400657,
+      "grad_norm": 1.6792598962783813,
+      "learning_rate": 0.0002,
+      "loss": 1.0988,
+      "step": 1460
+    },
+    {
+      "epoch": 4.827586206896552,
+      "grad_norm": 1.502026915550232,
+      "learning_rate": 0.0002,
+      "loss": 1.0871,
+      "step": 1470
+    },
+    {
+      "epoch": 4.860426929392447,
+      "grad_norm": 1.6605278253555298,
+      "learning_rate": 0.0002,
+      "loss": 1.0908,
+      "step": 1480
+    },
+    {
+      "epoch": 4.893267651888341,
+      "grad_norm": 1.9696134328842163,
+      "learning_rate": 0.0002,
+      "loss": 1.1198,
+      "step": 1490
+    },
+    {
+      "epoch": 4.926108374384237,
+      "grad_norm": 1.7231285572052002,
+      "learning_rate": 0.0002,
+      "loss": 1.0681,
+      "step": 1500
+    },
+    {
+      "epoch": 4.958949096880131,
+      "grad_norm": 1.4651379585266113,
+      "learning_rate": 0.0002,
+      "loss": 1.1278,
+      "step": 1510
+    },
+    {
+      "epoch": 4.991789819376026,
+      "grad_norm": 1.6087716817855835,
+      "learning_rate": 0.0002,
+      "loss": 1.1276,
+      "step": 1520
+    },
+    {
+      "epoch": 4.998357963875205,
+      "eval_loss": 2.104314088821411,
+      "eval_runtime": 149.1321,
+      "eval_samples_per_second": 3.574,
+      "eval_steps_per_second": 0.449,
+      "step": 1522
+    }
+  ],
+  "logging_steps": 10,
+  "max_steps": 2432,
+  "num_input_tokens_seen": 0,
+  "num_train_epochs": 8,
+  "save_steps": 200,
+  "stateful_callbacks": {
+    "TrainerControl": {
+      "args": {
+        "should_epoch_stop": false,
+        "should_evaluate": false,
+        "should_log": false,
+        "should_save": true,
+        "should_training_stop": false
+      },
+      "attributes": {}
+    }
+  },
+  "total_flos": 6.6799271411712e+16,
+  "train_batch_size": 1,
+  "trial_name": null,
+  "trial_params": null
+}

Mistral-7B-Instruct-v0.3_int4_hellaswag-routerbench-0shot_lr-0.0002_e-8_seq-512_lora-a-32-d-0.05-r-64_bs-1_gas-2_tf32-True_tunedata-portion-p-0.2-num-1405-sd-4/checkpoint-1522/training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:726dbd1b57eddad44e9f6bd0029e07b38acb3489720ae7c0f0fc0c9df3587bab
+size 5560

Mistral-7B-Instruct-v0.3_int4_hellaswag-routerbench-0shot_lr-0.0002_e-8_seq-512_lora-a-32-d-0.05-r-64_bs-1_gas-2_tf32-True_tunedata-portion-p-0.2-num-1405-sd-4/checkpoint-1827/README.md

@@ -0,0 +1,202 @@
+---
+base_model: mistralai/Mistral-7B-Instruct-v0.3
+library_name: peft
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.13.1

Mistral-7B-Instruct-v0.3_int4_hellaswag-routerbench-0shot_lr-0.0002_e-8_seq-512_lora-a-32-d-0.05-r-64_bs-1_gas-2_tf32-True_tunedata-portion-p-0.2-num-1405-sd-4/checkpoint-1827/adapter_config.json

@@ -0,0 +1,29 @@
+{
+  "alpha_pattern": {},
+  "auto_mapping": null,
+  "base_model_name_or_path": "mistralai/Mistral-7B-Instruct-v0.3",
+  "bias": "none",
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 32,
+  "lora_dropout": 0.05,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "r": 64,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "q_proj",
+    "v_proj"
+  ],
+  "task_type": "CAUSAL_LM",
+  "use_dora": false,
+  "use_rslora": false
+}

Mistral-7B-Instruct-v0.3_int4_hellaswag-routerbench-0shot_lr-0.0002_e-8_seq-512_lora-a-32-d-0.05-r-64_bs-1_gas-2_tf32-True_tunedata-portion-p-0.2-num-1405-sd-4/checkpoint-1827/adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0faf7336e0264d4f5240a36dc5b2c4ba9ca8806621870ebf6b4bc7cd0c61e627
+size 109069176

Mistral-7B-Instruct-v0.3_int4_hellaswag-routerbench-0shot_lr-0.0002_e-8_seq-512_lora-a-32-d-0.05-r-64_bs-1_gas-2_tf32-True_tunedata-portion-p-0.2-num-1405-sd-4/checkpoint-1827/optimizer.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e4ae786e4e7dc3050ad4aef7d0be0e958dc13ba5e439f077e29d62c0510358a8
+size 55532666

Mistral-7B-Instruct-v0.3_int4_hellaswag-routerbench-0shot_lr-0.0002_e-8_seq-512_lora-a-32-d-0.05-r-64_bs-1_gas-2_tf32-True_tunedata-portion-p-0.2-num-1405-sd-4/checkpoint-1827/rng_state.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e7382951b12e817108f42f80ade1b9604d643adfcfe1e9273f67b667d25b3f9b
+size 14244

Mistral-7B-Instruct-v0.3_int4_hellaswag-routerbench-0shot_lr-0.0002_e-8_seq-512_lora-a-32-d-0.05-r-64_bs-1_gas-2_tf32-True_tunedata-portion-p-0.2-num-1405-sd-4/checkpoint-1827/scheduler.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:38841519e925109736ce493a81a32baf57374f07cc4493ecbc110955926f5fa2
+size 1064

Mistral-7B-Instruct-v0.3_int4_hellaswag-routerbench-0shot_lr-0.0002_e-8_seq-512_lora-a-32-d-0.05-r-64_bs-1_gas-2_tf32-True_tunedata-portion-p-0.2-num-1405-sd-4/checkpoint-1827/special_tokens_map.json

@@ -0,0 +1,24 @@
+{
+  "bos_token": {
+    "content": "<s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "</s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": "</s>",
+  "unk_token": {
+    "content": "<unk>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

Mistral-7B-Instruct-v0.3_int4_hellaswag-routerbench-0shot_lr-0.0002_e-8_seq-512_lora-a-32-d-0.05-r-64_bs-1_gas-2_tf32-True_tunedata-portion-p-0.2-num-1405-sd-4/checkpoint-1827/tokenizer.model

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:37f00374dea48658ee8f5d0f21895b9bc55cb0103939607c8185bfd1c6ca1f89
+size 587404

Mistral-7B-Instruct-v0.3_int4_hellaswag-routerbench-0shot_lr-0.0002_e-8_seq-512_lora-a-32-d-0.05-r-64_bs-1_gas-2_tf32-True_tunedata-portion-p-0.2-num-1405-sd-4/checkpoint-1827/trainer_state.json

@@ -0,0 +1,1355 @@
+{
+  "best_metric": 1.790574550628662,
+  "best_model_checkpoint": "outputs-001/Mistral-7B-Instruct-v0.3_int4_hellaswag-routerbench-0shot_lr-0.0002_e-8_seq-512_lora-a-32-d-0.05-r-64_bs-1_gas-2_tf32-True_tunedata-portion-p-0.2-num-1405-sd-4/checkpoint-304",
+  "epoch": 6.0,
+  "eval_steps": 10,
+  "global_step": 1827,
+  "is_hyper_param_search": false,
+  "is_local_process_zero": true,
+  "is_world_process_zero": true,
+  "log_history": [
+    {
+      "epoch": 0.03284072249589491,
+      "grad_norm": 0.8926695585250854,
+      "learning_rate": 0.0002,
+      "loss": 2.7246,
+      "step": 10
+    },
+    {
+      "epoch": 0.06568144499178982,
+      "grad_norm": 1.181358814239502,
+      "learning_rate": 0.0002,
+      "loss": 2.1913,
+      "step": 20
+    },
+    {
+      "epoch": 0.09852216748768473,
+      "grad_norm": 0.7392292618751526,
+      "learning_rate": 0.0002,
+      "loss": 2.1022,
+      "step": 30
+    },
+    {
+      "epoch": 0.13136288998357964,
+      "grad_norm": 0.7944067716598511,
+      "learning_rate": 0.0002,
+      "loss": 2.071,
+      "step": 40
+    },
+    {
+      "epoch": 0.16420361247947454,
+      "grad_norm": 0.7858729958534241,
+      "learning_rate": 0.0002,
+      "loss": 2.0564,
+      "step": 50
+    },
+    {
+      "epoch": 0.19704433497536947,
+      "grad_norm": 0.7574328184127808,
+      "learning_rate": 0.0002,
+      "loss": 2.065,
+      "step": 60
+    },
+    {
+      "epoch": 0.22988505747126436,
+      "grad_norm": 0.694824755191803,
+      "learning_rate": 0.0002,
+      "loss": 1.9014,
+      "step": 70
+    },
+    {
+      "epoch": 0.2627257799671593,
+      "grad_norm": 0.6516484022140503,
+      "learning_rate": 0.0002,
+      "loss": 1.885,
+      "step": 80
+    },
+    {
+      "epoch": 0.2955665024630542,
+      "grad_norm": 0.6870642304420471,
+      "learning_rate": 0.0002,
+      "loss": 1.8555,
+      "step": 90
+    },
+    {
+      "epoch": 0.3284072249589491,
+      "grad_norm": 0.7598554491996765,
+      "learning_rate": 0.0002,
+      "loss": 1.84,
+      "step": 100
+    },
+    {
+      "epoch": 0.361247947454844,
+      "grad_norm": 0.6050140857696533,
+      "learning_rate": 0.0002,
+      "loss": 1.8204,
+      "step": 110
+    },
+    {
+      "epoch": 0.39408866995073893,
+      "grad_norm": 0.7373790740966797,
+      "learning_rate": 0.0002,
+      "loss": 1.8294,
+      "step": 120
+    },
+    {
+      "epoch": 0.4269293924466338,
+      "grad_norm": 0.6588838696479797,
+      "learning_rate": 0.0002,
+      "loss": 1.7682,
+      "step": 130
+    },
+    {
+      "epoch": 0.45977011494252873,
+      "grad_norm": 0.5314497947692871,
+      "learning_rate": 0.0002,
+      "loss": 1.7925,
+      "step": 140
+    },
+    {
+      "epoch": 0.49261083743842365,
+      "grad_norm": 0.6273255944252014,
+      "learning_rate": 0.0002,
+      "loss": 1.7992,
+      "step": 150
+    },
+    {
+      "epoch": 0.5254515599343186,
+      "grad_norm": 0.5744606256484985,
+      "learning_rate": 0.0002,
+      "loss": 1.8207,
+      "step": 160
+    },
+    {
+      "epoch": 0.5582922824302134,
+      "grad_norm": 0.577231228351593,
+      "learning_rate": 0.0002,
+      "loss": 1.8215,
+      "step": 170
+    },
+    {
+      "epoch": 0.5911330049261084,
+      "grad_norm": 0.5391182899475098,
+      "learning_rate": 0.0002,
+      "loss": 1.8125,
+      "step": 180
+    },
+    {
+      "epoch": 0.6239737274220033,
+      "grad_norm": 0.5863330364227295,
+      "learning_rate": 0.0002,
+      "loss": 1.8501,
+      "step": 190
+    },
+    {
+      "epoch": 0.6568144499178982,
+      "grad_norm": 0.6266984343528748,
+      "learning_rate": 0.0002,
+      "loss": 1.8055,
+      "step": 200
+    },
+    {
+      "epoch": 0.6896551724137931,
+      "grad_norm": 0.5842767357826233,
+      "learning_rate": 0.0002,
+      "loss": 1.7923,
+      "step": 210
+    },
+    {
+      "epoch": 0.722495894909688,
+      "grad_norm": 0.5255483984947205,
+      "learning_rate": 0.0002,
+      "loss": 1.7906,
+      "step": 220
+    },
+    {
+      "epoch": 0.7553366174055829,
+      "grad_norm": 0.7259632349014282,
+      "learning_rate": 0.0002,
+      "loss": 1.786,
+      "step": 230
+    },
+    {
+      "epoch": 0.7881773399014779,
+      "grad_norm": 0.5593494176864624,
+      "learning_rate": 0.0002,
+      "loss": 1.7291,
+      "step": 240
+    },
+    {
+      "epoch": 0.8210180623973727,
+      "grad_norm": 0.5614203214645386,
+      "learning_rate": 0.0002,
+      "loss": 1.8098,
+      "step": 250
+    },
+    {
+      "epoch": 0.8538587848932676,
+      "grad_norm": 0.6784001588821411,
+      "learning_rate": 0.0002,
+      "loss": 1.814,
+      "step": 260
+    },
+    {
+      "epoch": 0.8866995073891626,
+      "grad_norm": 0.5510041117668152,
+      "learning_rate": 0.0002,
+      "loss": 1.6856,
+      "step": 270
+    },
+    {
+      "epoch": 0.9195402298850575,
+      "grad_norm": 0.5453769564628601,
+      "learning_rate": 0.0002,
+      "loss": 1.7939,
+      "step": 280
+    },
+    {
+      "epoch": 0.9523809523809523,
+      "grad_norm": 0.5494107604026794,
+      "learning_rate": 0.0002,
+      "loss": 1.7489,
+      "step": 290
+    },
+    {
+      "epoch": 0.9852216748768473,
+      "grad_norm": 0.5768024921417236,
+      "learning_rate": 0.0002,
+      "loss": 1.8442,
+      "step": 300
+    },
+    {
+      "epoch": 0.9983579638752053,
+      "eval_loss": 1.790574550628662,
+      "eval_runtime": 147.7829,
+      "eval_samples_per_second": 3.607,
+      "eval_steps_per_second": 0.453,
+      "step": 304
+    },
+    {
+      "epoch": 1.0180623973727423,
+      "grad_norm": 0.5931118130683899,
+      "learning_rate": 0.0002,
+      "loss": 1.7061,
+      "step": 310
+    },
+    {
+      "epoch": 1.0509031198686372,
+      "grad_norm": 0.6433814764022827,
+      "learning_rate": 0.0002,
+      "loss": 1.6997,
+      "step": 320
+    },
+    {
+      "epoch": 1.083743842364532,
+      "grad_norm": 0.6801510453224182,
+      "learning_rate": 0.0002,
+      "loss": 1.697,
+      "step": 330
+    },
+    {
+      "epoch": 1.116584564860427,
+      "grad_norm": 0.6511064171791077,
+      "learning_rate": 0.0002,
+      "loss": 1.721,
+      "step": 340
+    },
+    {
+      "epoch": 1.1494252873563218,
+      "grad_norm": 0.6265677213668823,
+      "learning_rate": 0.0002,
+      "loss": 1.7209,
+      "step": 350
+    },
+    {
+      "epoch": 1.1822660098522166,
+      "grad_norm": 0.6211821436882019,
+      "learning_rate": 0.0002,
+      "loss": 1.6524,
+      "step": 360
+    },
+    {
+      "epoch": 1.2151067323481117,
+      "grad_norm": 0.6931810975074768,
+      "learning_rate": 0.0002,
+      "loss": 1.7091,
+      "step": 370
+    },
+    {
+      "epoch": 1.2479474548440066,
+      "grad_norm": 0.5978460311889648,
+      "learning_rate": 0.0002,
+      "loss": 1.6604,
+      "step": 380
+    },
+    {
+      "epoch": 1.2807881773399015,
+      "grad_norm": 0.6581379175186157,
+      "learning_rate": 0.0002,
+      "loss": 1.6707,
+      "step": 390
+    },
+    {
+      "epoch": 1.3136288998357963,
+      "grad_norm": 0.6794891953468323,
+      "learning_rate": 0.0002,
+      "loss": 1.6543,
+      "step": 400
+    },
+    {
+      "epoch": 1.3464696223316914,
+      "grad_norm": 0.6028420329093933,
+      "learning_rate": 0.0002,
+      "loss": 1.7261,
+      "step": 410
+    },
+    {
+      "epoch": 1.3793103448275863,
+      "grad_norm": 0.8568810820579529,
+      "learning_rate": 0.0002,
+      "loss": 1.7234,
+      "step": 420
+    },
+    {
+      "epoch": 1.4121510673234812,
+      "grad_norm": 0.6654542088508606,
+      "learning_rate": 0.0002,
+      "loss": 1.6885,
+      "step": 430
+    },
+    {
+      "epoch": 1.444991789819376,
+      "grad_norm": 0.6363505721092224,
+      "learning_rate": 0.0002,
+      "loss": 1.7335,
+      "step": 440
+    },
+    {
+      "epoch": 1.477832512315271,
+      "grad_norm": 0.7018159627914429,
+      "learning_rate": 0.0002,
+      "loss": 1.6947,
+      "step": 450
+    },
+    {
+      "epoch": 1.5106732348111658,
+      "grad_norm": 0.6718074679374695,
+      "learning_rate": 0.0002,
+      "loss": 1.7006,
+      "step": 460
+    },
+    {
+      "epoch": 1.5435139573070606,
+      "grad_norm": 0.6456331610679626,
+      "learning_rate": 0.0002,
+      "loss": 1.6504,
+      "step": 470
+    },
+    {
+      "epoch": 1.5763546798029555,
+      "grad_norm": 0.7088801860809326,
+      "learning_rate": 0.0002,
+      "loss": 1.6953,
+      "step": 480
+    },
+    {
+      "epoch": 1.6091954022988506,
+      "grad_norm": 0.8001429438591003,
+      "learning_rate": 0.0002,
+      "loss": 1.6521,
+      "step": 490
+    },
+    {
+      "epoch": 1.6420361247947455,
+      "grad_norm": 0.6954358816146851,
+      "learning_rate": 0.0002,
+      "loss": 1.6384,
+      "step": 500
+    },
+    {
+      "epoch": 1.6748768472906403,
+      "grad_norm": 0.6385519504547119,
+      "learning_rate": 0.0002,
+      "loss": 1.6606,
+      "step": 510
+    },
+    {
+      "epoch": 1.7077175697865354,
+      "grad_norm": 0.6736445426940918,
+      "learning_rate": 0.0002,
+      "loss": 1.7151,
+      "step": 520
+    },
+    {
+      "epoch": 1.7405582922824303,
+      "grad_norm": 0.6794690489768982,
+      "learning_rate": 0.0002,
+      "loss": 1.7363,
+      "step": 530
+    },
+    {
+      "epoch": 1.7733990147783252,
+      "grad_norm": 0.8341553211212158,
+      "learning_rate": 0.0002,
+      "loss": 1.6945,
+      "step": 540
+    },
+    {
+      "epoch": 1.80623973727422,
+      "grad_norm": 0.6141380071640015,
+      "learning_rate": 0.0002,
+      "loss": 1.6601,
+      "step": 550
+    },
+    {
+      "epoch": 1.839080459770115,
+      "grad_norm": 0.6553818583488464,
+      "learning_rate": 0.0002,
+      "loss": 1.6916,
+      "step": 560
+    },
+    {
+      "epoch": 1.8719211822660098,
+      "grad_norm": 0.675327718257904,
+      "learning_rate": 0.0002,
+      "loss": 1.681,
+      "step": 570
+    },
+    {
+      "epoch": 1.9047619047619047,
+      "grad_norm": 0.6748024821281433,
+      "learning_rate": 0.0002,
+      "loss": 1.6508,
+      "step": 580
+    },
+    {
+      "epoch": 1.9376026272577995,
+      "grad_norm": 0.7145724892616272,
+      "learning_rate": 0.0002,
+      "loss": 1.7898,
+      "step": 590
+    },
+    {
+      "epoch": 1.9704433497536946,
+      "grad_norm": 0.7319425940513611,
+      "learning_rate": 0.0002,
+      "loss": 1.7138,
+      "step": 600
+    },
+    {
+      "epoch": 2.0,
+      "eval_loss": 1.7979766130447388,
+      "eval_runtime": 147.9359,
+      "eval_samples_per_second": 3.603,
+      "eval_steps_per_second": 0.453,
+      "step": 609
+    },
+    {
+      "epoch": 2.0032840722495897,
+      "grad_norm": 0.5940006375312805,
+      "learning_rate": 0.0002,
+      "loss": 1.6249,
+      "step": 610
+    },
+    {
+      "epoch": 2.0361247947454846,
+      "grad_norm": 0.8300226330757141,
+      "learning_rate": 0.0002,
+      "loss": 1.5343,
+      "step": 620
+    },
+    {
+      "epoch": 2.0689655172413794,
+      "grad_norm": 0.7499504089355469,
+      "learning_rate": 0.0002,
+      "loss": 1.5071,
+      "step": 630
+    },
+    {
+      "epoch": 2.1018062397372743,
+      "grad_norm": 0.7384152412414551,
+      "learning_rate": 0.0002,
+      "loss": 1.5293,
+      "step": 640
+    },
+    {
+      "epoch": 2.134646962233169,
+      "grad_norm": 0.7212202548980713,
+      "learning_rate": 0.0002,
+      "loss": 1.4843,
+      "step": 650
+    },
+    {
+      "epoch": 2.167487684729064,
+      "grad_norm": 0.8744934797286987,
+      "learning_rate": 0.0002,
+      "loss": 1.4661,
+      "step": 660
+    },
+    {
+      "epoch": 2.200328407224959,
+      "grad_norm": 0.8432527184486389,
+      "learning_rate": 0.0002,
+      "loss": 1.4922,
+      "step": 670
+    },
+    {
+      "epoch": 2.233169129720854,
+      "grad_norm": 0.8430032730102539,
+      "learning_rate": 0.0002,
+      "loss": 1.4602,
+      "step": 680
+    },
+    {
+      "epoch": 2.2660098522167487,
+      "grad_norm": 0.8931224942207336,
+      "learning_rate": 0.0002,
+      "loss": 1.5109,
+      "step": 690
+    },
+    {
+      "epoch": 2.2988505747126435,
+      "grad_norm": 0.7747994065284729,
+      "learning_rate": 0.0002,
+      "loss": 1.4834,
+      "step": 700
+    },
+    {
+      "epoch": 2.3316912972085384,
+      "grad_norm": 0.839187502861023,
+      "learning_rate": 0.0002,
+      "loss": 1.5328,
+      "step": 710
+    },
+    {
+      "epoch": 2.3645320197044333,
+      "grad_norm": 0.8716842532157898,
+      "learning_rate": 0.0002,
+      "loss": 1.5716,
+      "step": 720
+    },
+    {
+      "epoch": 2.3973727422003286,
+      "grad_norm": 0.9200440049171448,
+      "learning_rate": 0.0002,
+      "loss": 1.5197,
+      "step": 730
+    },
+    {
+      "epoch": 2.4302134646962235,
+      "grad_norm": 0.7892614603042603,
+      "learning_rate": 0.0002,
+      "loss": 1.5512,
+      "step": 740
+    },
+    {
+      "epoch": 2.4630541871921183,
+      "grad_norm": 0.8656499981880188,
+      "learning_rate": 0.0002,
+      "loss": 1.4936,
+      "step": 750
+    },
+    {
+      "epoch": 2.495894909688013,
+      "grad_norm": 1.0396990776062012,
+      "learning_rate": 0.0002,
+      "loss": 1.5198,
+      "step": 760
+    },
+    {
+      "epoch": 2.528735632183908,
+      "grad_norm": 0.8359261751174927,
+      "learning_rate": 0.0002,
+      "loss": 1.5026,
+      "step": 770
+    },
+    {
+      "epoch": 2.561576354679803,
+      "grad_norm": 0.793073296546936,
+      "learning_rate": 0.0002,
+      "loss": 1.5454,
+      "step": 780
+    },
+    {
+      "epoch": 2.594417077175698,
+      "grad_norm": 0.9727579355239868,
+      "learning_rate": 0.0002,
+      "loss": 1.5254,
+      "step": 790
+    },
+    {
+      "epoch": 2.6272577996715927,
+      "grad_norm": 0.8486084342002869,
+      "learning_rate": 0.0002,
+      "loss": 1.5317,
+      "step": 800
+    },
+    {
+      "epoch": 2.6600985221674875,
+      "grad_norm": 0.8329479694366455,
+      "learning_rate": 0.0002,
+      "loss": 1.4904,
+      "step": 810
+    },
+    {
+      "epoch": 2.692939244663383,
+      "grad_norm": 0.8098273873329163,
+      "learning_rate": 0.0002,
+      "loss": 1.5422,
+      "step": 820
+    },
+    {
+      "epoch": 2.7257799671592773,
+      "grad_norm": 0.899185836315155,
+      "learning_rate": 0.0002,
+      "loss": 1.4642,
+      "step": 830
+    },
+    {
+      "epoch": 2.7586206896551726,
+      "grad_norm": 1.03939950466156,
+      "learning_rate": 0.0002,
+      "loss": 1.5208,
+      "step": 840
+    },
+    {
+      "epoch": 2.7914614121510675,
+      "grad_norm": 0.9275932908058167,
+      "learning_rate": 0.0002,
+      "loss": 1.5117,
+      "step": 850
+    },
+    {
+      "epoch": 2.8243021346469623,
+      "grad_norm": 0.8601354956626892,
+      "learning_rate": 0.0002,
+      "loss": 1.5891,
+      "step": 860
+    },
+    {
+      "epoch": 2.857142857142857,
+      "grad_norm": 0.8605263233184814,
+      "learning_rate": 0.0002,
+      "loss": 1.5398,
+      "step": 870
+    },
+    {
+      "epoch": 2.889983579638752,
+      "grad_norm": 2.0008926391601562,
+      "learning_rate": 0.0002,
+      "loss": 1.5964,
+      "step": 880
+    },
+    {
+      "epoch": 2.922824302134647,
+      "grad_norm": 0.8891443610191345,
+      "learning_rate": 0.0002,
+      "loss": 1.5407,
+      "step": 890
+    },
+    {
+      "epoch": 2.955665024630542,
+      "grad_norm": 0.9993823766708374,
+      "learning_rate": 0.0002,
+      "loss": 1.5517,
+      "step": 900
+    },
+    {
+      "epoch": 2.9885057471264367,
+      "grad_norm": 1.07162344455719,
+      "learning_rate": 0.0002,
+      "loss": 1.4897,
+      "step": 910
+    },
+    {
+      "epoch": 2.9983579638752054,
+      "eval_loss": 1.8474907875061035,
+      "eval_runtime": 148.1815,
+      "eval_samples_per_second": 3.597,
+      "eval_steps_per_second": 0.452,
+      "step": 913
+    },
+    {
+      "epoch": 3.0213464696223316,
+      "grad_norm": 1.0293525457382202,
+      "learning_rate": 0.0002,
+      "loss": 1.4149,
+      "step": 920
+    },
+    {
+      "epoch": 3.0541871921182264,
+      "grad_norm": 1.1002044677734375,
+      "learning_rate": 0.0002,
+      "loss": 1.2622,
+      "step": 930
+    },
+    {
+      "epoch": 3.0870279146141213,
+      "grad_norm": 0.9848016500473022,
+      "learning_rate": 0.0002,
+      "loss": 1.23,
+      "step": 940
+    },
+    {
+      "epoch": 3.1198686371100166,
+      "grad_norm": 1.195257306098938,
+      "learning_rate": 0.0002,
+      "loss": 1.2574,
+      "step": 950
+    },
+    {
+      "epoch": 3.1527093596059115,
+      "grad_norm": 1.416608452796936,
+      "learning_rate": 0.0002,
+      "loss": 1.3146,
+      "step": 960
+    },
+    {
+      "epoch": 3.1855500821018063,
+      "grad_norm": 1.1547739505767822,
+      "learning_rate": 0.0002,
+      "loss": 1.2348,
+      "step": 970
+    },
+    {
+      "epoch": 3.218390804597701,
+      "grad_norm": 0.9784232974052429,
+      "learning_rate": 0.0002,
+      "loss": 1.261,
+      "step": 980
+    },
+    {
+      "epoch": 3.251231527093596,
+      "grad_norm": 1.1947113275527954,
+      "learning_rate": 0.0002,
+      "loss": 1.2592,
+      "step": 990
+    },
+    {
+      "epoch": 3.284072249589491,
+      "grad_norm": 1.3581703901290894,
+      "learning_rate": 0.0002,
+      "loss": 1.278,
+      "step": 1000
+    },
+    {
+      "epoch": 3.316912972085386,
+      "grad_norm": 1.1447371244430542,
+      "learning_rate": 0.0002,
+      "loss": 1.2678,
+      "step": 1010
+    },
+    {
+      "epoch": 3.3497536945812807,
+      "grad_norm": 1.3974782228469849,
+      "learning_rate": 0.0002,
+      "loss": 1.3061,
+      "step": 1020
+    },
+    {
+      "epoch": 3.3825944170771756,
+      "grad_norm": 1.5755161046981812,
+      "learning_rate": 0.0002,
+      "loss": 1.2237,
+      "step": 1030
+    },
+    {
+      "epoch": 3.4154351395730704,
+      "grad_norm": 1.2665208578109741,
+      "learning_rate": 0.0002,
+      "loss": 1.2724,
+      "step": 1040
+    },
+    {
+      "epoch": 3.4482758620689653,
+      "grad_norm": 1.1338576078414917,
+      "learning_rate": 0.0002,
+      "loss": 1.3611,
+      "step": 1050
+    },
+    {
+      "epoch": 3.4811165845648606,
+      "grad_norm": 1.179924488067627,
+      "learning_rate": 0.0002,
+      "loss": 1.2958,
+      "step": 1060
+    },
+    {
+      "epoch": 3.5139573070607555,
+      "grad_norm": 1.55759596824646,
+      "learning_rate": 0.0002,
+      "loss": 1.3161,
+      "step": 1070
+    },
+    {
+      "epoch": 3.5467980295566504,
+      "grad_norm": 1.143595576286316,
+      "learning_rate": 0.0002,
+      "loss": 1.3307,
+      "step": 1080
+    },
+    {
+      "epoch": 3.5796387520525452,
+      "grad_norm": 1.1458665132522583,
+      "learning_rate": 0.0002,
+      "loss": 1.3719,
+      "step": 1090
+    },
+    {
+      "epoch": 3.61247947454844,
+      "grad_norm": 1.0880578756332397,
+      "learning_rate": 0.0002,
+      "loss": 1.3359,
+      "step": 1100
+    },
+    {
+      "epoch": 3.645320197044335,
+      "grad_norm": 1.2723114490509033,
+      "learning_rate": 0.0002,
+      "loss": 1.3686,
+      "step": 1110
+    },
+    {
+      "epoch": 3.67816091954023,
+      "grad_norm": 1.9975095987319946,
+      "learning_rate": 0.0002,
+      "loss": 1.3555,
+      "step": 1120
+    },
+    {
+      "epoch": 3.7110016420361247,
+      "grad_norm": 1.3761720657348633,
+      "learning_rate": 0.0002,
+      "loss": 1.2782,
+      "step": 1130
+    },
+    {
+      "epoch": 3.7438423645320196,
+      "grad_norm": 1.2107709646224976,
+      "learning_rate": 0.0002,
+      "loss": 1.3613,
+      "step": 1140
+    },
+    {
+      "epoch": 3.776683087027915,
+      "grad_norm": 1.16817307472229,
+      "learning_rate": 0.0002,
+      "loss": 1.3707,
+      "step": 1150
+    },
+    {
+      "epoch": 3.8095238095238093,
+      "grad_norm": 1.2264677286148071,
+      "learning_rate": 0.0002,
+      "loss": 1.3055,
+      "step": 1160
+    },
+    {
+      "epoch": 3.8423645320197046,
+      "grad_norm": 1.2869375944137573,
+      "learning_rate": 0.0002,
+      "loss": 1.3441,
+      "step": 1170
+    },
+    {
+      "epoch": 3.8752052545155995,
+      "grad_norm": 1.8852620124816895,
+      "learning_rate": 0.0002,
+      "loss": 1.3593,
+      "step": 1180
+    },
+    {
+      "epoch": 3.9080459770114944,
+      "grad_norm": 1.0901968479156494,
+      "learning_rate": 0.0002,
+      "loss": 1.3691,
+      "step": 1190
+    },
+    {
+      "epoch": 3.9408866995073892,
+      "grad_norm": 1.130061388015747,
+      "learning_rate": 0.0002,
+      "loss": 1.2799,
+      "step": 1200
+    },
+    {
+      "epoch": 3.973727422003284,
+      "grad_norm": 1.267082929611206,
+      "learning_rate": 0.0002,
+      "loss": 1.3157,
+      "step": 1210
+    },
+    {
+      "epoch": 4.0,
+      "eval_loss": 1.9435356855392456,
+      "eval_runtime": 152.8355,
+      "eval_samples_per_second": 3.487,
+      "eval_steps_per_second": 0.438,
+      "step": 1218
+    },
+    {
+      "epoch": 4.006568144499179,
+      "grad_norm": 1.1177356243133545,
+      "learning_rate": 0.0002,
+      "loss": 1.2551,
+      "step": 1220
+    },
+    {
+      "epoch": 4.039408866995074,
+      "grad_norm": 1.30461585521698,
+      "learning_rate": 0.0002,
+      "loss": 1.0096,
+      "step": 1230
+    },
+    {
+      "epoch": 4.072249589490969,
+      "grad_norm": 1.863768219947815,
+      "learning_rate": 0.0002,
+      "loss": 0.9757,
+      "step": 1240
+    },
+    {
+      "epoch": 4.105090311986864,
+      "grad_norm": 1.4688764810562134,
+      "learning_rate": 0.0002,
+      "loss": 1.1466,
+      "step": 1250
+    },
+    {
+      "epoch": 4.137931034482759,
+      "grad_norm": 1.3299670219421387,
+      "learning_rate": 0.0002,
+      "loss": 1.1102,
+      "step": 1260
+    },
+    {
+      "epoch": 4.170771756978653,
+      "grad_norm": 1.726919412612915,
+      "learning_rate": 0.0002,
+      "loss": 0.9916,
+      "step": 1270
+    },
+    {
+      "epoch": 4.203612479474549,
+      "grad_norm": 1.6402113437652588,
+      "learning_rate": 0.0002,
+      "loss": 1.137,
+      "step": 1280
+    },
+    {
+      "epoch": 4.236453201970443,
+      "grad_norm": 1.6000759601593018,
+      "learning_rate": 0.0002,
+      "loss": 1.0551,
+      "step": 1290
+    },
+    {
+      "epoch": 4.269293924466338,
+      "grad_norm": 1.4904356002807617,
+      "learning_rate": 0.0002,
+      "loss": 1.1378,
+      "step": 1300
+    },
+    {
+      "epoch": 4.302134646962233,
+      "grad_norm": 1.8119593858718872,
+      "learning_rate": 0.0002,
+      "loss": 1.0761,
+      "step": 1310
+    },
+    {
+      "epoch": 4.334975369458128,
+      "grad_norm": 1.5564337968826294,
+      "learning_rate": 0.0002,
+      "loss": 1.0773,
+      "step": 1320
+    },
+    {
+      "epoch": 4.3678160919540225,
+      "grad_norm": 1.6291178464889526,
+      "learning_rate": 0.0002,
+      "loss": 1.0441,
+      "step": 1330
+    },
+    {
+      "epoch": 4.400656814449918,
+      "grad_norm": 1.691028356552124,
+      "learning_rate": 0.0002,
+      "loss": 1.0053,
+      "step": 1340
+    },
+    {
+      "epoch": 4.433497536945813,
+      "grad_norm": 1.7010000944137573,
+      "learning_rate": 0.0002,
+      "loss": 1.035,
+      "step": 1350
+    },
+    {
+      "epoch": 4.466338259441708,
+      "grad_norm": 2.038245439529419,
+      "learning_rate": 0.0002,
+      "loss": 1.0319,
+      "step": 1360
+    },
+    {
+      "epoch": 4.499178981937603,
+      "grad_norm": 1.5663248300552368,
+      "learning_rate": 0.0002,
+      "loss": 1.0681,
+      "step": 1370
+    },
+    {
+      "epoch": 4.532019704433497,
+      "grad_norm": 1.6198927164077759,
+      "learning_rate": 0.0002,
+      "loss": 1.1214,
+      "step": 1380
+    },
+    {
+      "epoch": 4.564860426929393,
+      "grad_norm": 1.53022301197052,
+      "learning_rate": 0.0002,
+      "loss": 1.0945,
+      "step": 1390
+    },
+    {
+      "epoch": 4.597701149425287,
+      "grad_norm": 1.4284729957580566,
+      "learning_rate": 0.0002,
+      "loss": 1.1352,
+      "step": 1400
+    },
+    {
+      "epoch": 4.630541871921182,
+      "grad_norm": 1.741274118423462,
+      "learning_rate": 0.0002,
+      "loss": 1.0917,
+      "step": 1410
+    },
+    {
+      "epoch": 4.663382594417077,
+      "grad_norm": 1.5251778364181519,
+      "learning_rate": 0.0002,
+      "loss": 1.081,
+      "step": 1420
+    },
+    {
+      "epoch": 4.696223316912972,
+      "grad_norm": 1.9084422588348389,
+      "learning_rate": 0.0002,
+      "loss": 1.0455,
+      "step": 1430
+    },
+    {
+      "epoch": 4.7290640394088665,
+      "grad_norm": 1.868015170097351,
+      "learning_rate": 0.0002,
+      "loss": 1.0751,
+      "step": 1440
+    },
+    {
+      "epoch": 4.761904761904762,
+      "grad_norm": 1.634352445602417,
+      "learning_rate": 0.0002,
+      "loss": 1.1199,
+      "step": 1450
+    },
+    {
+      "epoch": 4.794745484400657,
+      "grad_norm": 1.6792598962783813,
+      "learning_rate": 0.0002,
+      "loss": 1.0988,
+      "step": 1460
+    },
+    {
+      "epoch": 4.827586206896552,
+      "grad_norm": 1.502026915550232,
+      "learning_rate": 0.0002,
+      "loss": 1.0871,
+      "step": 1470
+    },
+    {
+      "epoch": 4.860426929392447,
+      "grad_norm": 1.6605278253555298,
+      "learning_rate": 0.0002,
+      "loss": 1.0908,
+      "step": 1480
+    },
+    {
+      "epoch": 4.893267651888341,
+      "grad_norm": 1.9696134328842163,
+      "learning_rate": 0.0002,
+      "loss": 1.1198,
+      "step": 1490
+    },
+    {
+      "epoch": 4.926108374384237,
+      "grad_norm": 1.7231285572052002,
+      "learning_rate": 0.0002,
+      "loss": 1.0681,
+      "step": 1500
+    },
+    {
+      "epoch": 4.958949096880131,
+      "grad_norm": 1.4651379585266113,
+      "learning_rate": 0.0002,
+      "loss": 1.1278,
+      "step": 1510
+    },
+    {
+      "epoch": 4.991789819376026,
+      "grad_norm": 1.6087716817855835,
+      "learning_rate": 0.0002,
+      "loss": 1.1276,
+      "step": 1520
+    },
+    {
+      "epoch": 4.998357963875205,
+      "eval_loss": 2.104314088821411,
+      "eval_runtime": 149.1321,
+      "eval_samples_per_second": 3.574,
+      "eval_steps_per_second": 0.449,
+      "step": 1522
+    },
+    {
+      "epoch": 5.024630541871921,
+      "grad_norm": 2.2468345165252686,
+      "learning_rate": 0.0002,
+      "loss": 0.8613,
+      "step": 1530
+    },
+    {
+      "epoch": 5.057471264367816,
+      "grad_norm": 1.7066363096237183,
+      "learning_rate": 0.0002,
+      "loss": 0.871,
+      "step": 1540
+    },
+    {
+      "epoch": 5.090311986863711,
+      "grad_norm": 1.8440189361572266,
+      "learning_rate": 0.0002,
+      "loss": 0.7936,
+      "step": 1550
+    },
+    {
+      "epoch": 5.123152709359606,
+      "grad_norm": 1.90162992477417,
+      "learning_rate": 0.0002,
+      "loss": 0.8204,
+      "step": 1560
+    },
+    {
+      "epoch": 5.155993431855501,
+      "grad_norm": 2.479116916656494,
+      "learning_rate": 0.0002,
+      "loss": 0.8448,
+      "step": 1570
+    },
+    {
+      "epoch": 5.188834154351396,
+      "grad_norm": 2.0990519523620605,
+      "learning_rate": 0.0002,
+      "loss": 0.8365,
+      "step": 1580
+    },
+    {
+      "epoch": 5.221674876847291,
+      "grad_norm": 2.1840813159942627,
+      "learning_rate": 0.0002,
+      "loss": 0.8216,
+      "step": 1590
+    },
+    {
+      "epoch": 5.254515599343185,
+      "grad_norm": 2.479950428009033,
+      "learning_rate": 0.0002,
+      "loss": 0.8651,
+      "step": 1600
+    },
+    {
+      "epoch": 5.287356321839081,
+      "grad_norm": 1.6248779296875,
+      "learning_rate": 0.0002,
+      "loss": 0.8056,
+      "step": 1610
+    },
+    {
+      "epoch": 5.320197044334975,
+      "grad_norm": 1.7853254079818726,
+      "learning_rate": 0.0002,
+      "loss": 0.8257,
+      "step": 1620
+    },
+    {
+      "epoch": 5.35303776683087,
+      "grad_norm": 2.062437057495117,
+      "learning_rate": 0.0002,
+      "loss": 0.806,
+      "step": 1630
+    },
+    {
+      "epoch": 5.385878489326765,
+      "grad_norm": 1.9853318929672241,
+      "learning_rate": 0.0002,
+      "loss": 0.8321,
+      "step": 1640
+    },
+    {
+      "epoch": 5.41871921182266,
+      "grad_norm": 2.03126859664917,
+      "learning_rate": 0.0002,
+      "loss": 0.8379,
+      "step": 1650
+    },
+    {
+      "epoch": 5.451559934318555,
+      "grad_norm": 2.133615016937256,
+      "learning_rate": 0.0002,
+      "loss": 0.8638,
+      "step": 1660
+    },
+    {
+      "epoch": 5.48440065681445,
+      "grad_norm": 1.9745090007781982,
+      "learning_rate": 0.0002,
+      "loss": 0.8668,
+      "step": 1670
+    },
+    {
+      "epoch": 5.517241379310345,
+      "grad_norm": 2.4828860759735107,
+      "learning_rate": 0.0002,
+      "loss": 0.884,
+      "step": 1680
+    },
+    {
+      "epoch": 5.55008210180624,
+      "grad_norm": 2.007389545440674,
+      "learning_rate": 0.0002,
+      "loss": 0.8866,
+      "step": 1690
+    },
+    {
+      "epoch": 5.582922824302135,
+      "grad_norm": 2.181378126144409,
+      "learning_rate": 0.0002,
+      "loss": 0.9247,
+      "step": 1700
+    },
+    {
+      "epoch": 5.615763546798029,
+      "grad_norm": 2.476907253265381,
+      "learning_rate": 0.0002,
+      "loss": 0.9077,
+      "step": 1710
+    },
+    {
+      "epoch": 5.648604269293925,
+      "grad_norm": 1.9788070917129517,
+      "learning_rate": 0.0002,
+      "loss": 0.9327,
+      "step": 1720
+    },
+    {
+      "epoch": 5.681444991789819,
+      "grad_norm": 2.259640693664551,
+      "learning_rate": 0.0002,
+      "loss": 0.8107,
+      "step": 1730
+    },
+    {
+      "epoch": 5.714285714285714,
+      "grad_norm": 2.1496458053588867,
+      "learning_rate": 0.0002,
+      "loss": 0.9101,
+      "step": 1740
+    },
+    {
+      "epoch": 5.747126436781609,
+      "grad_norm": 1.9823484420776367,
+      "learning_rate": 0.0002,
+      "loss": 0.854,
+      "step": 1750
+    },
+    {
+      "epoch": 5.779967159277504,
+      "grad_norm": 2.473726511001587,
+      "learning_rate": 0.0002,
+      "loss": 0.8648,
+      "step": 1760
+    },
+    {
+      "epoch": 5.812807881773399,
+      "grad_norm": 1.9412373304367065,
+      "learning_rate": 0.0002,
+      "loss": 0.9335,
+      "step": 1770
+    },
+    {
+      "epoch": 5.845648604269294,
+      "grad_norm": 1.963869333267212,
+      "learning_rate": 0.0002,
+      "loss": 0.8586,
+      "step": 1780
+    },
+    {
+      "epoch": 5.878489326765189,
+      "grad_norm": 2.3070645332336426,
+      "learning_rate": 0.0002,
+      "loss": 0.8671,
+      "step": 1790
+    },
+    {
+      "epoch": 5.911330049261084,
+      "grad_norm": 1.8040424585342407,
+      "learning_rate": 0.0002,
+      "loss": 0.9002,
+      "step": 1800
+    },
+    {
+      "epoch": 5.944170771756979,
+      "grad_norm": 1.9620094299316406,
+      "learning_rate": 0.0002,
+      "loss": 0.8802,
+      "step": 1810
+    },
+    {
+      "epoch": 5.977011494252873,
+      "grad_norm": 2.0748379230499268,
+      "learning_rate": 0.0002,
+      "loss": 0.8473,
+      "step": 1820
+    },
+    {
+      "epoch": 6.0,
+      "eval_loss": 2.3067190647125244,
+      "eval_runtime": 156.1886,
+      "eval_samples_per_second": 3.413,
+      "eval_steps_per_second": 0.429,
+      "step": 1827
+    }
+  ],
+  "logging_steps": 10,
+  "max_steps": 2432,
+  "num_input_tokens_seen": 0,
+  "num_train_epochs": 8,
+  "save_steps": 200,
+  "stateful_callbacks": {
+    "TrainerControl": {
+      "args": {
+        "should_epoch_stop": false,
+        "should_evaluate": false,
+        "should_log": false,
+        "should_save": true,
+        "should_training_stop": false
+      },
+      "attributes": {}
+    }
+  },
+  "total_flos": 8.01591256940544e+16,
+  "train_batch_size": 1,
+  "trial_name": null,
+  "trial_params": null
+}

Mistral-7B-Instruct-v0.3_int4_hellaswag-routerbench-0shot_lr-0.0002_e-8_seq-512_lora-a-32-d-0.05-r-64_bs-1_gas-2_tf32-True_tunedata-portion-p-0.2-num-1405-sd-4/checkpoint-1827/training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:726dbd1b57eddad44e9f6bd0029e07b38acb3489720ae7c0f0fc0c9df3587bab
+size 5560

Mistral-7B-Instruct-v0.3_int4_hellaswag-routerbench-0shot_lr-0.0002_e-8_seq-512_lora-a-32-d-0.05-r-64_bs-1_gas-2_tf32-True_tunedata-portion-p-0.2-num-1405-sd-4/checkpoint-2131/README.md

@@ -0,0 +1,202 @@
+---
+base_model: mistralai/Mistral-7B-Instruct-v0.3
+library_name: peft
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.13.1

Mistral-7B-Instruct-v0.3_int4_hellaswag-routerbench-0shot_lr-0.0002_e-8_seq-512_lora-a-32-d-0.05-r-64_bs-1_gas-2_tf32-True_tunedata-portion-p-0.2-num-1405-sd-4/checkpoint-2131/adapter_config.json

@@ -0,0 +1,29 @@
+{
+  "alpha_pattern": {},
+  "auto_mapping": null,
+  "base_model_name_or_path": "mistralai/Mistral-7B-Instruct-v0.3",
+  "bias": "none",
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 32,
+  "lora_dropout": 0.05,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "r": 64,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "q_proj",
+    "v_proj"
+  ],
+  "task_type": "CAUSAL_LM",
+  "use_dora": false,
+  "use_rslora": false
+}

Mistral-7B-Instruct-v0.3_int4_hellaswag-routerbench-0shot_lr-0.0002_e-8_seq-512_lora-a-32-d-0.05-r-64_bs-1_gas-2_tf32-True_tunedata-portion-p-0.2-num-1405-sd-4/checkpoint-2131/adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:759765974f5f450c39c7bc08eb81cecc6be4fdba773bb2b61400077ffb3659c8
+size 109069176

Mistral-7B-Instruct-v0.3_int4_hellaswag-routerbench-0shot_lr-0.0002_e-8_seq-512_lora-a-32-d-0.05-r-64_bs-1_gas-2_tf32-True_tunedata-portion-p-0.2-num-1405-sd-4/checkpoint-2131/optimizer.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:16ba87af9cd9b49b6af651fdeeb18c10fb28dfee98286b100c281a40d993d21c
+size 55532666

Mistral-7B-Instruct-v0.3_int4_hellaswag-routerbench-0shot_lr-0.0002_e-8_seq-512_lora-a-32-d-0.05-r-64_bs-1_gas-2_tf32-True_tunedata-portion-p-0.2-num-1405-sd-4/checkpoint-2131/rng_state.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6e076337959bf0f0bceedab1370f98153dc25f3975378da47a8b425550a9d615
+size 14244

Mistral-7B-Instruct-v0.3_int4_hellaswag-routerbench-0shot_lr-0.0002_e-8_seq-512_lora-a-32-d-0.05-r-64_bs-1_gas-2_tf32-True_tunedata-portion-p-0.2-num-1405-sd-4/checkpoint-2131/scheduler.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f8573ba5704b05cd8f56b6c84bc7bb9f77eb449ed7bd9cc918e412a1ae12db44
+size 1064

Mistral-7B-Instruct-v0.3_int4_hellaswag-routerbench-0shot_lr-0.0002_e-8_seq-512_lora-a-32-d-0.05-r-64_bs-1_gas-2_tf32-True_tunedata-portion-p-0.2-num-1405-sd-4/checkpoint-2131/special_tokens_map.json

@@ -0,0 +1,24 @@
+{
+  "bos_token": {
+    "content": "<s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "</s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": "</s>",
+  "unk_token": {
+    "content": "<unk>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

Mistral-7B-Instruct-v0.3_int4_hellaswag-routerbench-0shot_lr-0.0002_e-8_seq-512_lora-a-32-d-0.05-r-64_bs-1_gas-2_tf32-True_tunedata-portion-p-0.2-num-1405-sd-4/checkpoint-2131/tokenizer.model

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:37f00374dea48658ee8f5d0f21895b9bc55cb0103939607c8185bfd1c6ca1f89
+size 587404

Mistral-7B-Instruct-v0.3_int4_hellaswag-routerbench-0shot_lr-0.0002_e-8_seq-512_lora-a-32-d-0.05-r-64_bs-1_gas-2_tf32-True_tunedata-portion-p-0.2-num-1405-sd-4/checkpoint-2131/trainer_state.json

@@ -0,0 +1,1580 @@
+{
+  "best_metric": 1.790574550628662,
+  "best_model_checkpoint": "outputs-001/Mistral-7B-Instruct-v0.3_int4_hellaswag-routerbench-0shot_lr-0.0002_e-8_seq-512_lora-a-32-d-0.05-r-64_bs-1_gas-2_tf32-True_tunedata-portion-p-0.2-num-1405-sd-4/checkpoint-304",
+  "epoch": 6.998357963875205,
+  "eval_steps": 10,
+  "global_step": 2131,
+  "is_hyper_param_search": false,
+  "is_local_process_zero": true,
+  "is_world_process_zero": true,
+  "log_history": [
+    {
+      "epoch": 0.03284072249589491,
+      "grad_norm": 0.8926695585250854,
+      "learning_rate": 0.0002,
+      "loss": 2.7246,
+      "step": 10
+    },
+    {
+      "epoch": 0.06568144499178982,
+      "grad_norm": 1.181358814239502,
+      "learning_rate": 0.0002,
+      "loss": 2.1913,
+      "step": 20
+    },
+    {
+      "epoch": 0.09852216748768473,
+      "grad_norm": 0.7392292618751526,
+      "learning_rate": 0.0002,
+      "loss": 2.1022,
+      "step": 30
+    },
+    {
+      "epoch": 0.13136288998357964,
+      "grad_norm": 0.7944067716598511,
+      "learning_rate": 0.0002,
+      "loss": 2.071,
+      "step": 40
+    },
+    {
+      "epoch": 0.16420361247947454,
+      "grad_norm": 0.7858729958534241,
+      "learning_rate": 0.0002,
+      "loss": 2.0564,
+      "step": 50
+    },
+    {
+      "epoch": 0.19704433497536947,
+      "grad_norm": 0.7574328184127808,
+      "learning_rate": 0.0002,
+      "loss": 2.065,
+      "step": 60
+    },
+    {
+      "epoch": 0.22988505747126436,
+      "grad_norm": 0.694824755191803,
+      "learning_rate": 0.0002,
+      "loss": 1.9014,
+      "step": 70
+    },
+    {
+      "epoch": 0.2627257799671593,
+      "grad_norm": 0.6516484022140503,
+      "learning_rate": 0.0002,
+      "loss": 1.885,
+      "step": 80
+    },
+    {
+      "epoch": 0.2955665024630542,
+      "grad_norm": 0.6870642304420471,
+      "learning_rate": 0.0002,
+      "loss": 1.8555,
+      "step": 90
+    },
+    {
+      "epoch": 0.3284072249589491,
+      "grad_norm": 0.7598554491996765,
+      "learning_rate": 0.0002,
+      "loss": 1.84,
+      "step": 100
+    },
+    {
+      "epoch": 0.361247947454844,
+      "grad_norm": 0.6050140857696533,
+      "learning_rate": 0.0002,
+      "loss": 1.8204,
+      "step": 110
+    },
+    {
+      "epoch": 0.39408866995073893,
+      "grad_norm": 0.7373790740966797,
+      "learning_rate": 0.0002,
+      "loss": 1.8294,
+      "step": 120
+    },
+    {
+      "epoch": 0.4269293924466338,
+      "grad_norm": 0.6588838696479797,
+      "learning_rate": 0.0002,
+      "loss": 1.7682,
+      "step": 130
+    },
+    {
+      "epoch": 0.45977011494252873,
+      "grad_norm": 0.5314497947692871,
+      "learning_rate": 0.0002,
+      "loss": 1.7925,
+      "step": 140
+    },
+    {
+      "epoch": 0.49261083743842365,
+      "grad_norm": 0.6273255944252014,
+      "learning_rate": 0.0002,
+      "loss": 1.7992,
+      "step": 150
+    },
+    {
+      "epoch": 0.5254515599343186,
+      "grad_norm": 0.5744606256484985,
+      "learning_rate": 0.0002,
+      "loss": 1.8207,
+      "step": 160
+    },
+    {
+      "epoch": 0.5582922824302134,
+      "grad_norm": 0.577231228351593,
+      "learning_rate": 0.0002,
+      "loss": 1.8215,
+      "step": 170
+    },
+    {
+      "epoch": 0.5911330049261084,
+      "grad_norm": 0.5391182899475098,
+      "learning_rate": 0.0002,
+      "loss": 1.8125,
+      "step": 180
+    },
+    {
+      "epoch": 0.6239737274220033,
+      "grad_norm": 0.5863330364227295,
+      "learning_rate": 0.0002,
+      "loss": 1.8501,
+      "step": 190
+    },
+    {
+      "epoch": 0.6568144499178982,
+      "grad_norm": 0.6266984343528748,
+      "learning_rate": 0.0002,
+      "loss": 1.8055,
+      "step": 200
+    },
+    {
+      "epoch": 0.6896551724137931,
+      "grad_norm": 0.5842767357826233,
+      "learning_rate": 0.0002,
+      "loss": 1.7923,
+      "step": 210
+    },
+    {
+      "epoch": 0.722495894909688,
+      "grad_norm": 0.5255483984947205,
+      "learning_rate": 0.0002,
+      "loss": 1.7906,
+      "step": 220
+    },
+    {
+      "epoch": 0.7553366174055829,
+      "grad_norm": 0.7259632349014282,
+      "learning_rate": 0.0002,
+      "loss": 1.786,
+      "step": 230
+    },
+    {
+      "epoch": 0.7881773399014779,
+      "grad_norm": 0.5593494176864624,
+      "learning_rate": 0.0002,
+      "loss": 1.7291,
+      "step": 240
+    },
+    {
+      "epoch": 0.8210180623973727,
+      "grad_norm": 0.5614203214645386,
+      "learning_rate": 0.0002,
+      "loss": 1.8098,
+      "step": 250
+    },
+    {
+      "epoch": 0.8538587848932676,
+      "grad_norm": 0.6784001588821411,
+      "learning_rate": 0.0002,
+      "loss": 1.814,
+      "step": 260
+    },
+    {
+      "epoch": 0.8866995073891626,
+      "grad_norm": 0.5510041117668152,
+      "learning_rate": 0.0002,
+      "loss": 1.6856,
+      "step": 270
+    },
+    {
+      "epoch": 0.9195402298850575,
+      "grad_norm": 0.5453769564628601,
+      "learning_rate": 0.0002,
+      "loss": 1.7939,
+      "step": 280
+    },
+    {
+      "epoch": 0.9523809523809523,
+      "grad_norm": 0.5494107604026794,
+      "learning_rate": 0.0002,
+      "loss": 1.7489,
+      "step": 290
+    },
+    {
+      "epoch": 0.9852216748768473,
+      "grad_norm": 0.5768024921417236,
+      "learning_rate": 0.0002,
+      "loss": 1.8442,
+      "step": 300
+    },
+    {
+      "epoch": 0.9983579638752053,
+      "eval_loss": 1.790574550628662,
+      "eval_runtime": 147.7829,
+      "eval_samples_per_second": 3.607,
+      "eval_steps_per_second": 0.453,
+      "step": 304
+    },
+    {
+      "epoch": 1.0180623973727423,
+      "grad_norm": 0.5931118130683899,
+      "learning_rate": 0.0002,
+      "loss": 1.7061,
+      "step": 310
+    },
+    {
+      "epoch": 1.0509031198686372,
+      "grad_norm": 0.6433814764022827,
+      "learning_rate": 0.0002,
+      "loss": 1.6997,
+      "step": 320
+    },
+    {
+      "epoch": 1.083743842364532,
+      "grad_norm": 0.6801510453224182,
+      "learning_rate": 0.0002,
+      "loss": 1.697,
+      "step": 330
+    },
+    {
+      "epoch": 1.116584564860427,
+      "grad_norm": 0.6511064171791077,
+      "learning_rate": 0.0002,
+      "loss": 1.721,
+      "step": 340
+    },
+    {
+      "epoch": 1.1494252873563218,
+      "grad_norm": 0.6265677213668823,
+      "learning_rate": 0.0002,
+      "loss": 1.7209,
+      "step": 350
+    },
+    {
+      "epoch": 1.1822660098522166,
+      "grad_norm": 0.6211821436882019,
+      "learning_rate": 0.0002,
+      "loss": 1.6524,
+      "step": 360
+    },
+    {
+      "epoch": 1.2151067323481117,
+      "grad_norm": 0.6931810975074768,
+      "learning_rate": 0.0002,
+      "loss": 1.7091,
+      "step": 370
+    },
+    {
+      "epoch": 1.2479474548440066,
+      "grad_norm": 0.5978460311889648,
+      "learning_rate": 0.0002,
+      "loss": 1.6604,
+      "step": 380
+    },
+    {
+      "epoch": 1.2807881773399015,
+      "grad_norm": 0.6581379175186157,
+      "learning_rate": 0.0002,
+      "loss": 1.6707,
+      "step": 390
+    },
+    {
+      "epoch": 1.3136288998357963,
+      "grad_norm": 0.6794891953468323,
+      "learning_rate": 0.0002,
+      "loss": 1.6543,
+      "step": 400
+    },
+    {
+      "epoch": 1.3464696223316914,
+      "grad_norm": 0.6028420329093933,
+      "learning_rate": 0.0002,
+      "loss": 1.7261,
+      "step": 410
+    },
+    {
+      "epoch": 1.3793103448275863,
+      "grad_norm": 0.8568810820579529,
+      "learning_rate": 0.0002,
+      "loss": 1.7234,
+      "step": 420
+    },
+    {
+      "epoch": 1.4121510673234812,
+      "grad_norm": 0.6654542088508606,
+      "learning_rate": 0.0002,
+      "loss": 1.6885,
+      "step": 430
+    },
+    {
+      "epoch": 1.444991789819376,
+      "grad_norm": 0.6363505721092224,
+      "learning_rate": 0.0002,
+      "loss": 1.7335,
+      "step": 440
+    },
+    {
+      "epoch": 1.477832512315271,
+      "grad_norm": 0.7018159627914429,
+      "learning_rate": 0.0002,
+      "loss": 1.6947,
+      "step": 450
+    },
+    {
+      "epoch": 1.5106732348111658,
+      "grad_norm": 0.6718074679374695,
+      "learning_rate": 0.0002,
+      "loss": 1.7006,
+      "step": 460
+    },
+    {
+      "epoch": 1.5435139573070606,
+      "grad_norm": 0.6456331610679626,
+      "learning_rate": 0.0002,
+      "loss": 1.6504,
+      "step": 470
+    },
+    {
+      "epoch": 1.5763546798029555,
+      "grad_norm": 0.7088801860809326,
+      "learning_rate": 0.0002,
+      "loss": 1.6953,
+      "step": 480
+    },
+    {
+      "epoch": 1.6091954022988506,
+      "grad_norm": 0.8001429438591003,
+      "learning_rate": 0.0002,
+      "loss": 1.6521,
+      "step": 490
+    },
+    {
+      "epoch": 1.6420361247947455,
+      "grad_norm": 0.6954358816146851,
+      "learning_rate": 0.0002,
+      "loss": 1.6384,
+      "step": 500
+    },
+    {
+      "epoch": 1.6748768472906403,
+      "grad_norm": 0.6385519504547119,
+      "learning_rate": 0.0002,
+      "loss": 1.6606,
+      "step": 510
+    },
+    {
+      "epoch": 1.7077175697865354,
+      "grad_norm": 0.6736445426940918,
+      "learning_rate": 0.0002,
+      "loss": 1.7151,
+      "step": 520
+    },
+    {
+      "epoch": 1.7405582922824303,
+      "grad_norm": 0.6794690489768982,
+      "learning_rate": 0.0002,
+      "loss": 1.7363,
+      "step": 530
+    },
+    {
+      "epoch": 1.7733990147783252,
+      "grad_norm": 0.8341553211212158,
+      "learning_rate": 0.0002,
+      "loss": 1.6945,
+      "step": 540
+    },
+    {
+      "epoch": 1.80623973727422,
+      "grad_norm": 0.6141380071640015,
+      "learning_rate": 0.0002,
+      "loss": 1.6601,
+      "step": 550
+    },
+    {
+      "epoch": 1.839080459770115,
+      "grad_norm": 0.6553818583488464,
+      "learning_rate": 0.0002,
+      "loss": 1.6916,
+      "step": 560
+    },
+    {
+      "epoch": 1.8719211822660098,
+      "grad_norm": 0.675327718257904,
+      "learning_rate": 0.0002,
+      "loss": 1.681,
+      "step": 570
+    },
+    {
+      "epoch": 1.9047619047619047,
+      "grad_norm": 0.6748024821281433,
+      "learning_rate": 0.0002,
+      "loss": 1.6508,
+      "step": 580
+    },
+    {
+      "epoch": 1.9376026272577995,
+      "grad_norm": 0.7145724892616272,
+      "learning_rate": 0.0002,
+      "loss": 1.7898,
+      "step": 590
+    },
+    {
+      "epoch": 1.9704433497536946,
+      "grad_norm": 0.7319425940513611,
+      "learning_rate": 0.0002,
+      "loss": 1.7138,
+      "step": 600
+    },
+    {
+      "epoch": 2.0,
+      "eval_loss": 1.7979766130447388,
+      "eval_runtime": 147.9359,
+      "eval_samples_per_second": 3.603,
+      "eval_steps_per_second": 0.453,
+      "step": 609
+    },
+    {
+      "epoch": 2.0032840722495897,
+      "grad_norm": 0.5940006375312805,
+      "learning_rate": 0.0002,
+      "loss": 1.6249,
+      "step": 610
+    },
+    {
+      "epoch": 2.0361247947454846,
+      "grad_norm": 0.8300226330757141,
+      "learning_rate": 0.0002,
+      "loss": 1.5343,
+      "step": 620
+    },
+    {
+      "epoch": 2.0689655172413794,
+      "grad_norm": 0.7499504089355469,
+      "learning_rate": 0.0002,
+      "loss": 1.5071,
+      "step": 630
+    },
+    {
+      "epoch": 2.1018062397372743,
+      "grad_norm": 0.7384152412414551,
+      "learning_rate": 0.0002,
+      "loss": 1.5293,
+      "step": 640
+    },
+    {
+      "epoch": 2.134646962233169,
+      "grad_norm": 0.7212202548980713,
+      "learning_rate": 0.0002,
+      "loss": 1.4843,
+      "step": 650
+    },
+    {
+      "epoch": 2.167487684729064,
+      "grad_norm": 0.8744934797286987,
+      "learning_rate": 0.0002,
+      "loss": 1.4661,
+      "step": 660
+    },
+    {
+      "epoch": 2.200328407224959,
+      "grad_norm": 0.8432527184486389,
+      "learning_rate": 0.0002,
+      "loss": 1.4922,
+      "step": 670
+    },
+    {
+      "epoch": 2.233169129720854,
+      "grad_norm": 0.8430032730102539,
+      "learning_rate": 0.0002,
+      "loss": 1.4602,
+      "step": 680
+    },
+    {
+      "epoch": 2.2660098522167487,
+      "grad_norm": 0.8931224942207336,
+      "learning_rate": 0.0002,
+      "loss": 1.5109,
+      "step": 690
+    },
+    {
+      "epoch": 2.2988505747126435,
+      "grad_norm": 0.7747994065284729,
+      "learning_rate": 0.0002,
+      "loss": 1.4834,
+      "step": 700
+    },
+    {
+      "epoch": 2.3316912972085384,
+      "grad_norm": 0.839187502861023,
+      "learning_rate": 0.0002,
+      "loss": 1.5328,
+      "step": 710
+    },
+    {
+      "epoch": 2.3645320197044333,
+      "grad_norm": 0.8716842532157898,
+      "learning_rate": 0.0002,
+      "loss": 1.5716,
+      "step": 720
+    },
+    {
+      "epoch": 2.3973727422003286,
+      "grad_norm": 0.9200440049171448,
+      "learning_rate": 0.0002,
+      "loss": 1.5197,
+      "step": 730
+    },
+    {
+      "epoch": 2.4302134646962235,
+      "grad_norm": 0.7892614603042603,
+      "learning_rate": 0.0002,
+      "loss": 1.5512,
+      "step": 740
+    },
+    {
+      "epoch": 2.4630541871921183,
+      "grad_norm": 0.8656499981880188,
+      "learning_rate": 0.0002,
+      "loss": 1.4936,
+      "step": 750
+    },
+    {
+      "epoch": 2.495894909688013,
+      "grad_norm": 1.0396990776062012,
+      "learning_rate": 0.0002,
+      "loss": 1.5198,
+      "step": 760
+    },
+    {
+      "epoch": 2.528735632183908,
+      "grad_norm": 0.8359261751174927,
+      "learning_rate": 0.0002,
+      "loss": 1.5026,
+      "step": 770
+    },
+    {
+      "epoch": 2.561576354679803,
+      "grad_norm": 0.793073296546936,
+      "learning_rate": 0.0002,
+      "loss": 1.5454,
+      "step": 780
+    },
+    {
+      "epoch": 2.594417077175698,
+      "grad_norm": 0.9727579355239868,
+      "learning_rate": 0.0002,
+      "loss": 1.5254,
+      "step": 790
+    },
+    {
+      "epoch": 2.6272577996715927,
+      "grad_norm": 0.8486084342002869,
+      "learning_rate": 0.0002,
+      "loss": 1.5317,
+      "step": 800
+    },
+    {
+      "epoch": 2.6600985221674875,
+      "grad_norm": 0.8329479694366455,
+      "learning_rate": 0.0002,
+      "loss": 1.4904,
+      "step": 810
+    },
+    {
+      "epoch": 2.692939244663383,
+      "grad_norm": 0.8098273873329163,
+      "learning_rate": 0.0002,
+      "loss": 1.5422,
+      "step": 820
+    },
+    {
+      "epoch": 2.7257799671592773,
+      "grad_norm": 0.899185836315155,
+      "learning_rate": 0.0002,
+      "loss": 1.4642,
+      "step": 830
+    },
+    {
+      "epoch": 2.7586206896551726,
+      "grad_norm": 1.03939950466156,
+      "learning_rate": 0.0002,
+      "loss": 1.5208,
+      "step": 840
+    },
+    {
+      "epoch": 2.7914614121510675,
+      "grad_norm": 0.9275932908058167,
+      "learning_rate": 0.0002,
+      "loss": 1.5117,
+      "step": 850
+    },
+    {
+      "epoch": 2.8243021346469623,
+      "grad_norm": 0.8601354956626892,
+      "learning_rate": 0.0002,
+      "loss": 1.5891,
+      "step": 860
+    },
+    {
+      "epoch": 2.857142857142857,
+      "grad_norm": 0.8605263233184814,
+      "learning_rate": 0.0002,
+      "loss": 1.5398,
+      "step": 870
+    },
+    {
+      "epoch": 2.889983579638752,
+      "grad_norm": 2.0008926391601562,
+      "learning_rate": 0.0002,
+      "loss": 1.5964,
+      "step": 880
+    },
+    {
+      "epoch": 2.922824302134647,
+      "grad_norm": 0.8891443610191345,
+      "learning_rate": 0.0002,
+      "loss": 1.5407,
+      "step": 890
+    },
+    {
+      "epoch": 2.955665024630542,
+      "grad_norm": 0.9993823766708374,
+      "learning_rate": 0.0002,
+      "loss": 1.5517,
+      "step": 900
+    },
+    {
+      "epoch": 2.9885057471264367,
+      "grad_norm": 1.07162344455719,
+      "learning_rate": 0.0002,
+      "loss": 1.4897,
+      "step": 910
+    },
+    {
+      "epoch": 2.9983579638752054,
+      "eval_loss": 1.8474907875061035,
+      "eval_runtime": 148.1815,
+      "eval_samples_per_second": 3.597,
+      "eval_steps_per_second": 0.452,
+      "step": 913
+    },
+    {
+      "epoch": 3.0213464696223316,
+      "grad_norm": 1.0293525457382202,
+      "learning_rate": 0.0002,
+      "loss": 1.4149,
+      "step": 920
+    },
+    {
+      "epoch": 3.0541871921182264,
+      "grad_norm": 1.1002044677734375,
+      "learning_rate": 0.0002,
+      "loss": 1.2622,
+      "step": 930
+    },
+    {
+      "epoch": 3.0870279146141213,
+      "grad_norm": 0.9848016500473022,
+      "learning_rate": 0.0002,
+      "loss": 1.23,
+      "step": 940
+    },
+    {
+      "epoch": 3.1198686371100166,
+      "grad_norm": 1.195257306098938,
+      "learning_rate": 0.0002,
+      "loss": 1.2574,
+      "step": 950
+    },
+    {
+      "epoch": 3.1527093596059115,
+      "grad_norm": 1.416608452796936,
+      "learning_rate": 0.0002,
+      "loss": 1.3146,
+      "step": 960
+    },
+    {
+      "epoch": 3.1855500821018063,
+      "grad_norm": 1.1547739505767822,
+      "learning_rate": 0.0002,
+      "loss": 1.2348,
+      "step": 970
+    },
+    {
+      "epoch": 3.218390804597701,
+      "grad_norm": 0.9784232974052429,
+      "learning_rate": 0.0002,
+      "loss": 1.261,
+      "step": 980
+    },
+    {
+      "epoch": 3.251231527093596,
+      "grad_norm": 1.1947113275527954,
+      "learning_rate": 0.0002,
+      "loss": 1.2592,
+      "step": 990
+    },
+    {
+      "epoch": 3.284072249589491,
+      "grad_norm": 1.3581703901290894,
+      "learning_rate": 0.0002,
+      "loss": 1.278,
+      "step": 1000
+    },
+    {
+      "epoch": 3.316912972085386,
+      "grad_norm": 1.1447371244430542,
+      "learning_rate": 0.0002,
+      "loss": 1.2678,
+      "step": 1010
+    },
+    {
+      "epoch": 3.3497536945812807,
+      "grad_norm": 1.3974782228469849,
+      "learning_rate": 0.0002,
+      "loss": 1.3061,
+      "step": 1020
+    },
+    {
+      "epoch": 3.3825944170771756,
+      "grad_norm": 1.5755161046981812,
+      "learning_rate": 0.0002,
+      "loss": 1.2237,
+      "step": 1030
+    },
+    {
+      "epoch": 3.4154351395730704,
+      "grad_norm": 1.2665208578109741,
+      "learning_rate": 0.0002,
+      "loss": 1.2724,
+      "step": 1040
+    },
+    {
+      "epoch": 3.4482758620689653,
+      "grad_norm": 1.1338576078414917,
+      "learning_rate": 0.0002,
+      "loss": 1.3611,
+      "step": 1050
+    },
+    {
+      "epoch": 3.4811165845648606,
+      "grad_norm": 1.179924488067627,
+      "learning_rate": 0.0002,
+      "loss": 1.2958,
+      "step": 1060
+    },
+    {
+      "epoch": 3.5139573070607555,
+      "grad_norm": 1.55759596824646,
+      "learning_rate": 0.0002,
+      "loss": 1.3161,
+      "step": 1070
+    },
+    {
+      "epoch": 3.5467980295566504,
+      "grad_norm": 1.143595576286316,
+      "learning_rate": 0.0002,
+      "loss": 1.3307,
+      "step": 1080
+    },
+    {
+      "epoch": 3.5796387520525452,
+      "grad_norm": 1.1458665132522583,
+      "learning_rate": 0.0002,
+      "loss": 1.3719,
+      "step": 1090
+    },
+    {
+      "epoch": 3.61247947454844,
+      "grad_norm": 1.0880578756332397,
+      "learning_rate": 0.0002,
+      "loss": 1.3359,
+      "step": 1100
+    },
+    {
+      "epoch": 3.645320197044335,
+      "grad_norm": 1.2723114490509033,
+      "learning_rate": 0.0002,
+      "loss": 1.3686,
+      "step": 1110
+    },
+    {
+      "epoch": 3.67816091954023,
+      "grad_norm": 1.9975095987319946,
+      "learning_rate": 0.0002,
+      "loss": 1.3555,
+      "step": 1120
+    },
+    {
+      "epoch": 3.7110016420361247,
+      "grad_norm": 1.3761720657348633,
+      "learning_rate": 0.0002,
+      "loss": 1.2782,
+      "step": 1130
+    },
+    {
+      "epoch": 3.7438423645320196,
+      "grad_norm": 1.2107709646224976,
+      "learning_rate": 0.0002,
+      "loss": 1.3613,
+      "step": 1140
+    },
+    {
+      "epoch": 3.776683087027915,
+      "grad_norm": 1.16817307472229,
+      "learning_rate": 0.0002,
+      "loss": 1.3707,
+      "step": 1150
+    },
+    {
+      "epoch": 3.8095238095238093,
+      "grad_norm": 1.2264677286148071,
+      "learning_rate": 0.0002,
+      "loss": 1.3055,
+      "step": 1160
+    },
+    {
+      "epoch": 3.8423645320197046,
+      "grad_norm": 1.2869375944137573,
+      "learning_rate": 0.0002,
+      "loss": 1.3441,
+      "step": 1170
+    },
+    {
+      "epoch": 3.8752052545155995,
+      "grad_norm": 1.8852620124816895,
+      "learning_rate": 0.0002,
+      "loss": 1.3593,
+      "step": 1180
+    },
+    {
+      "epoch": 3.9080459770114944,
+      "grad_norm": 1.0901968479156494,
+      "learning_rate": 0.0002,
+      "loss": 1.3691,
+      "step": 1190
+    },
+    {
+      "epoch": 3.9408866995073892,
+      "grad_norm": 1.130061388015747,
+      "learning_rate": 0.0002,
+      "loss": 1.2799,
+      "step": 1200
+    },
+    {
+      "epoch": 3.973727422003284,
+      "grad_norm": 1.267082929611206,
+      "learning_rate": 0.0002,
+      "loss": 1.3157,
+      "step": 1210
+    },
+    {
+      "epoch": 4.0,
+      "eval_loss": 1.9435356855392456,
+      "eval_runtime": 152.8355,
+      "eval_samples_per_second": 3.487,
+      "eval_steps_per_second": 0.438,
+      "step": 1218
+    },
+    {
+      "epoch": 4.006568144499179,
+      "grad_norm": 1.1177356243133545,
+      "learning_rate": 0.0002,
+      "loss": 1.2551,
+      "step": 1220
+    },
+    {
+      "epoch": 4.039408866995074,
+      "grad_norm": 1.30461585521698,
+      "learning_rate": 0.0002,
+      "loss": 1.0096,
+      "step": 1230
+    },
+    {
+      "epoch": 4.072249589490969,
+      "grad_norm": 1.863768219947815,
+      "learning_rate": 0.0002,
+      "loss": 0.9757,
+      "step": 1240
+    },
+    {
+      "epoch": 4.105090311986864,
+      "grad_norm": 1.4688764810562134,
+      "learning_rate": 0.0002,
+      "loss": 1.1466,
+      "step": 1250
+    },
+    {
+      "epoch": 4.137931034482759,
+      "grad_norm": 1.3299670219421387,
+      "learning_rate": 0.0002,
+      "loss": 1.1102,
+      "step": 1260
+    },
+    {
+      "epoch": 4.170771756978653,
+      "grad_norm": 1.726919412612915,
+      "learning_rate": 0.0002,
+      "loss": 0.9916,
+      "step": 1270
+    },
+    {
+      "epoch": 4.203612479474549,
+      "grad_norm": 1.6402113437652588,
+      "learning_rate": 0.0002,
+      "loss": 1.137,
+      "step": 1280
+    },
+    {
+      "epoch": 4.236453201970443,
+      "grad_norm": 1.6000759601593018,
+      "learning_rate": 0.0002,
+      "loss": 1.0551,
+      "step": 1290
+    },
+    {
+      "epoch": 4.269293924466338,
+      "grad_norm": 1.4904356002807617,
+      "learning_rate": 0.0002,
+      "loss": 1.1378,
+      "step": 1300
+    },
+    {
+      "epoch": 4.302134646962233,
+      "grad_norm": 1.8119593858718872,
+      "learning_rate": 0.0002,
+      "loss": 1.0761,
+      "step": 1310
+    },
+    {
+      "epoch": 4.334975369458128,
+      "grad_norm": 1.5564337968826294,
+      "learning_rate": 0.0002,
+      "loss": 1.0773,
+      "step": 1320
+    },
+    {
+      "epoch": 4.3678160919540225,
+      "grad_norm": 1.6291178464889526,
+      "learning_rate": 0.0002,
+      "loss": 1.0441,
+      "step": 1330
+    },
+    {
+      "epoch": 4.400656814449918,
+      "grad_norm": 1.691028356552124,
+      "learning_rate": 0.0002,
+      "loss": 1.0053,
+      "step": 1340
+    },
+    {
+      "epoch": 4.433497536945813,
+      "grad_norm": 1.7010000944137573,
+      "learning_rate": 0.0002,
+      "loss": 1.035,
+      "step": 1350
+    },
+    {
+      "epoch": 4.466338259441708,
+      "grad_norm": 2.038245439529419,
+      "learning_rate": 0.0002,
+      "loss": 1.0319,
+      "step": 1360
+    },
+    {
+      "epoch": 4.499178981937603,
+      "grad_norm": 1.5663248300552368,
+      "learning_rate": 0.0002,
+      "loss": 1.0681,
+      "step": 1370
+    },
+    {
+      "epoch": 4.532019704433497,
+      "grad_norm": 1.6198927164077759,
+      "learning_rate": 0.0002,
+      "loss": 1.1214,
+      "step": 1380
+    },
+    {
+      "epoch": 4.564860426929393,
+      "grad_norm": 1.53022301197052,
+      "learning_rate": 0.0002,
+      "loss": 1.0945,
+      "step": 1390
+    },
+    {
+      "epoch": 4.597701149425287,
+      "grad_norm": 1.4284729957580566,
+      "learning_rate": 0.0002,
+      "loss": 1.1352,
+      "step": 1400
+    },
+    {
+      "epoch": 4.630541871921182,
+      "grad_norm": 1.741274118423462,
+      "learning_rate": 0.0002,
+      "loss": 1.0917,
+      "step": 1410
+    },
+    {
+      "epoch": 4.663382594417077,
+      "grad_norm": 1.5251778364181519,
+      "learning_rate": 0.0002,
+      "loss": 1.081,
+      "step": 1420
+    },
+    {
+      "epoch": 4.696223316912972,
+      "grad_norm": 1.9084422588348389,
+      "learning_rate": 0.0002,
+      "loss": 1.0455,
+      "step": 1430
+    },
+    {
+      "epoch": 4.7290640394088665,
+      "grad_norm": 1.868015170097351,
+      "learning_rate": 0.0002,
+      "loss": 1.0751,
+      "step": 1440
+    },
+    {
+      "epoch": 4.761904761904762,
+      "grad_norm": 1.634352445602417,
+      "learning_rate": 0.0002,
+      "loss": 1.1199,
+      "step": 1450
+    },
+    {
+      "epoch": 4.794745484400657,
+      "grad_norm": 1.6792598962783813,
+      "learning_rate": 0.0002,
+      "loss": 1.0988,
+      "step": 1460
+    },
+    {
+      "epoch": 4.827586206896552,
+      "grad_norm": 1.502026915550232,
+      "learning_rate": 0.0002,
+      "loss": 1.0871,
+      "step": 1470
+    },
+    {
+      "epoch": 4.860426929392447,
+      "grad_norm": 1.6605278253555298,
+      "learning_rate": 0.0002,
+      "loss": 1.0908,
+      "step": 1480
+    },
+    {
+      "epoch": 4.893267651888341,
+      "grad_norm": 1.9696134328842163,
+      "learning_rate": 0.0002,
+      "loss": 1.1198,
+      "step": 1490
+    },
+    {
+      "epoch": 4.926108374384237,
+      "grad_norm": 1.7231285572052002,
+      "learning_rate": 0.0002,
+      "loss": 1.0681,
+      "step": 1500
+    },
+    {
+      "epoch": 4.958949096880131,
+      "grad_norm": 1.4651379585266113,
+      "learning_rate": 0.0002,
+      "loss": 1.1278,
+      "step": 1510
+    },
+    {
+      "epoch": 4.991789819376026,
+      "grad_norm": 1.6087716817855835,
+      "learning_rate": 0.0002,
+      "loss": 1.1276,
+      "step": 1520
+    },
+    {
+      "epoch": 4.998357963875205,
+      "eval_loss": 2.104314088821411,
+      "eval_runtime": 149.1321,
+      "eval_samples_per_second": 3.574,
+      "eval_steps_per_second": 0.449,
+      "step": 1522
+    },
+    {
+      "epoch": 5.024630541871921,
+      "grad_norm": 2.2468345165252686,
+      "learning_rate": 0.0002,
+      "loss": 0.8613,
+      "step": 1530
+    },
+    {
+      "epoch": 5.057471264367816,
+      "grad_norm": 1.7066363096237183,
+      "learning_rate": 0.0002,
+      "loss": 0.871,
+      "step": 1540
+    },
+    {
+      "epoch": 5.090311986863711,
+      "grad_norm": 1.8440189361572266,
+      "learning_rate": 0.0002,
+      "loss": 0.7936,
+      "step": 1550
+    },
+    {
+      "epoch": 5.123152709359606,
+      "grad_norm": 1.90162992477417,
+      "learning_rate": 0.0002,
+      "loss": 0.8204,
+      "step": 1560
+    },
+    {
+      "epoch": 5.155993431855501,
+      "grad_norm": 2.479116916656494,
+      "learning_rate": 0.0002,
+      "loss": 0.8448,
+      "step": 1570
+    },
+    {
+      "epoch": 5.188834154351396,
+      "grad_norm": 2.0990519523620605,
+      "learning_rate": 0.0002,
+      "loss": 0.8365,
+      "step": 1580
+    },
+    {
+      "epoch": 5.221674876847291,
+      "grad_norm": 2.1840813159942627,
+      "learning_rate": 0.0002,
+      "loss": 0.8216,
+      "step": 1590
+    },
+    {
+      "epoch": 5.254515599343185,
+      "grad_norm": 2.479950428009033,
+      "learning_rate": 0.0002,
+      "loss": 0.8651,
+      "step": 1600
+    },
+    {
+      "epoch": 5.287356321839081,
+      "grad_norm": 1.6248779296875,
+      "learning_rate": 0.0002,
+      "loss": 0.8056,
+      "step": 1610
+    },
+    {
+      "epoch": 5.320197044334975,
+      "grad_norm": 1.7853254079818726,
+      "learning_rate": 0.0002,
+      "loss": 0.8257,
+      "step": 1620
+    },
+    {
+      "epoch": 5.35303776683087,
+      "grad_norm": 2.062437057495117,
+      "learning_rate": 0.0002,
+      "loss": 0.806,
+      "step": 1630
+    },
+    {
+      "epoch": 5.385878489326765,
+      "grad_norm": 1.9853318929672241,
+      "learning_rate": 0.0002,
+      "loss": 0.8321,
+      "step": 1640
+    },
+    {
+      "epoch": 5.41871921182266,
+      "grad_norm": 2.03126859664917,
+      "learning_rate": 0.0002,
+      "loss": 0.8379,
+      "step": 1650
+    },
+    {
+      "epoch": 5.451559934318555,
+      "grad_norm": 2.133615016937256,
+      "learning_rate": 0.0002,
+      "loss": 0.8638,
+      "step": 1660
+    },
+    {
+      "epoch": 5.48440065681445,
+      "grad_norm": 1.9745090007781982,
+      "learning_rate": 0.0002,
+      "loss": 0.8668,
+      "step": 1670
+    },
+    {
+      "epoch": 5.517241379310345,
+      "grad_norm": 2.4828860759735107,
+      "learning_rate": 0.0002,
+      "loss": 0.884,
+      "step": 1680
+    },
+    {
+      "epoch": 5.55008210180624,
+      "grad_norm": 2.007389545440674,
+      "learning_rate": 0.0002,
+      "loss": 0.8866,
+      "step": 1690
+    },
+    {
+      "epoch": 5.582922824302135,
+      "grad_norm": 2.181378126144409,
+      "learning_rate": 0.0002,
+      "loss": 0.9247,
+      "step": 1700
+    },
+    {
+      "epoch": 5.615763546798029,
+      "grad_norm": 2.476907253265381,
+      "learning_rate": 0.0002,
+      "loss": 0.9077,
+      "step": 1710
+    },
+    {
+      "epoch": 5.648604269293925,
+      "grad_norm": 1.9788070917129517,
+      "learning_rate": 0.0002,
+      "loss": 0.9327,
+      "step": 1720
+    },
+    {
+      "epoch": 5.681444991789819,
+      "grad_norm": 2.259640693664551,
+      "learning_rate": 0.0002,
+      "loss": 0.8107,
+      "step": 1730
+    },
+    {
+      "epoch": 5.714285714285714,
+      "grad_norm": 2.1496458053588867,
+      "learning_rate": 0.0002,
+      "loss": 0.9101,
+      "step": 1740
+    },
+    {
+      "epoch": 5.747126436781609,
+      "grad_norm": 1.9823484420776367,
+      "learning_rate": 0.0002,
+      "loss": 0.854,
+      "step": 1750
+    },
+    {
+      "epoch": 5.779967159277504,
+      "grad_norm": 2.473726511001587,
+      "learning_rate": 0.0002,
+      "loss": 0.8648,
+      "step": 1760
+    },
+    {
+      "epoch": 5.812807881773399,
+      "grad_norm": 1.9412373304367065,
+      "learning_rate": 0.0002,
+      "loss": 0.9335,
+      "step": 1770
+    },
+    {
+      "epoch": 5.845648604269294,
+      "grad_norm": 1.963869333267212,
+      "learning_rate": 0.0002,
+      "loss": 0.8586,
+      "step": 1780
+    },
+    {
+      "epoch": 5.878489326765189,
+      "grad_norm": 2.3070645332336426,
+      "learning_rate": 0.0002,
+      "loss": 0.8671,
+      "step": 1790
+    },
+    {
+      "epoch": 5.911330049261084,
+      "grad_norm": 1.8040424585342407,
+      "learning_rate": 0.0002,
+      "loss": 0.9002,
+      "step": 1800
+    },
+    {
+      "epoch": 5.944170771756979,
+      "grad_norm": 1.9620094299316406,
+      "learning_rate": 0.0002,
+      "loss": 0.8802,
+      "step": 1810
+    },
+    {
+      "epoch": 5.977011494252873,
+      "grad_norm": 2.0748379230499268,
+      "learning_rate": 0.0002,
+      "loss": 0.8473,
+      "step": 1820
+    },
+    {
+      "epoch": 6.0,
+      "eval_loss": 2.3067190647125244,
+      "eval_runtime": 156.1886,
+      "eval_samples_per_second": 3.413,
+      "eval_steps_per_second": 0.429,
+      "step": 1827
+    },
+    {
+      "epoch": 6.009852216748769,
+      "grad_norm": 1.9004411697387695,
+      "learning_rate": 0.0002,
+      "loss": 0.8317,
+      "step": 1830
+    },
+    {
+      "epoch": 6.042692939244663,
+      "grad_norm": 2.71014404296875,
+      "learning_rate": 0.0002,
+      "loss": 0.6321,
+      "step": 1840
+    },
+    {
+      "epoch": 6.075533661740558,
+      "grad_norm": 1.7549858093261719,
+      "learning_rate": 0.0002,
+      "loss": 0.5904,
+      "step": 1850
+    },
+    {
+      "epoch": 6.108374384236453,
+      "grad_norm": 2.2088961601257324,
+      "learning_rate": 0.0002,
+      "loss": 0.5983,
+      "step": 1860
+    },
+    {
+      "epoch": 6.141215106732348,
+      "grad_norm": 2.2269537448883057,
+      "learning_rate": 0.0002,
+      "loss": 0.6765,
+      "step": 1870
+    },
+    {
+      "epoch": 6.174055829228243,
+      "grad_norm": 2.8383333683013916,
+      "learning_rate": 0.0002,
+      "loss": 0.655,
+      "step": 1880
+    },
+    {
+      "epoch": 6.206896551724138,
+      "grad_norm": 2.300685405731201,
+      "learning_rate": 0.0002,
+      "loss": 0.6488,
+      "step": 1890
+    },
+    {
+      "epoch": 6.239737274220033,
+      "grad_norm": 2.548427104949951,
+      "learning_rate": 0.0002,
+      "loss": 0.6573,
+      "step": 1900
+    },
+    {
+      "epoch": 6.272577996715928,
+      "grad_norm": 2.5417912006378174,
+      "learning_rate": 0.0002,
+      "loss": 0.6172,
+      "step": 1910
+    },
+    {
+      "epoch": 6.305418719211823,
+      "grad_norm": 3.31034779548645,
+      "learning_rate": 0.0002,
+      "loss": 0.6207,
+      "step": 1920
+    },
+    {
+      "epoch": 6.338259441707717,
+      "grad_norm": 2.3062500953674316,
+      "learning_rate": 0.0002,
+      "loss": 0.65,
+      "step": 1930
+    },
+    {
+      "epoch": 6.371100164203613,
+      "grad_norm": 2.603363037109375,
+      "learning_rate": 0.0002,
+      "loss": 0.6362,
+      "step": 1940
+    },
+    {
+      "epoch": 6.403940886699507,
+      "grad_norm": 2.1587536334991455,
+      "learning_rate": 0.0002,
+      "loss": 0.7101,
+      "step": 1950
+    },
+    {
+      "epoch": 6.436781609195402,
+      "grad_norm": 2.197301149368286,
+      "learning_rate": 0.0002,
+      "loss": 0.7088,
+      "step": 1960
+    },
+    {
+      "epoch": 6.469622331691297,
+      "grad_norm": 2.361581802368164,
+      "learning_rate": 0.0002,
+      "loss": 0.6812,
+      "step": 1970
+    },
+    {
+      "epoch": 6.502463054187192,
+      "grad_norm": 2.243354558944702,
+      "learning_rate": 0.0002,
+      "loss": 0.6467,
+      "step": 1980
+    },
+    {
+      "epoch": 6.535303776683087,
+      "grad_norm": 2.3370299339294434,
+      "learning_rate": 0.0002,
+      "loss": 0.6857,
+      "step": 1990
+    },
+    {
+      "epoch": 6.568144499178982,
+      "grad_norm": 2.3022608757019043,
+      "learning_rate": 0.0002,
+      "loss": 0.6637,
+      "step": 2000
+    },
+    {
+      "epoch": 6.600985221674877,
+      "grad_norm": 2.754183769226074,
+      "learning_rate": 0.0002,
+      "loss": 0.6749,
+      "step": 2010
+    },
+    {
+      "epoch": 6.633825944170772,
+      "grad_norm": 2.340019702911377,
+      "learning_rate": 0.0002,
+      "loss": 0.7157,
+      "step": 2020
+    },
+    {
+      "epoch": 6.666666666666667,
+      "grad_norm": 2.9439103603363037,
+      "learning_rate": 0.0002,
+      "loss": 0.6851,
+      "step": 2030
+    },
+    {
+      "epoch": 6.699507389162561,
+      "grad_norm": 2.188605308532715,
+      "learning_rate": 0.0002,
+      "loss": 0.6907,
+      "step": 2040
+    },
+    {
+      "epoch": 6.732348111658457,
+      "grad_norm": 2.420973777770996,
+      "learning_rate": 0.0002,
+      "loss": 0.6558,
+      "step": 2050
+    },
+    {
+      "epoch": 6.765188834154351,
+      "grad_norm": 2.4219534397125244,
+      "learning_rate": 0.0002,
+      "loss": 0.6984,
+      "step": 2060
+    },
+    {
+      "epoch": 6.798029556650246,
+      "grad_norm": 2.357070207595825,
+      "learning_rate": 0.0002,
+      "loss": 0.6625,
+      "step": 2070
+    },
+    {
+      "epoch": 6.830870279146141,
+      "grad_norm": 2.480423927307129,
+      "learning_rate": 0.0002,
+      "loss": 0.6841,
+      "step": 2080
+    },
+    {
+      "epoch": 6.863711001642036,
+      "grad_norm": 2.5054664611816406,
+      "learning_rate": 0.0002,
+      "loss": 0.7173,
+      "step": 2090
+    },
+    {
+      "epoch": 6.896551724137931,
+      "grad_norm": 2.0294229984283447,
+      "learning_rate": 0.0002,
+      "loss": 0.6838,
+      "step": 2100
+    },
+    {
+      "epoch": 6.929392446633826,
+      "grad_norm": 2.240886926651001,
+      "learning_rate": 0.0002,
+      "loss": 0.718,
+      "step": 2110
+    },
+    {
+      "epoch": 6.962233169129721,
+      "grad_norm": 2.7301197052001953,
+      "learning_rate": 0.0002,
+      "loss": 0.718,
+      "step": 2120
+    },
+    {
+      "epoch": 6.995073891625616,
+      "grad_norm": 2.1306824684143066,
+      "learning_rate": 0.0002,
+      "loss": 0.659,
+      "step": 2130
+    },
+    {
+      "epoch": 6.998357963875205,
+      "eval_loss": 2.562674045562744,
+      "eval_runtime": 156.2717,
+      "eval_samples_per_second": 3.411,
+      "eval_steps_per_second": 0.429,
+      "step": 2131
+    }
+  ],
+  "logging_steps": 10,
+  "max_steps": 2432,
+  "num_input_tokens_seen": 0,
+  "num_train_epochs": 8,
+  "save_steps": 200,
+  "stateful_callbacks": {
+    "TrainerControl": {
+      "args": {
+        "should_epoch_stop": false,
+        "should_evaluate": false,
+        "should_log": false,
+        "should_save": true,
+        "should_training_stop": false
+      },
+      "attributes": {}
+    }
+  },
+  "total_flos": 9.35189799763968e+16,
+  "train_batch_size": 1,
+  "trial_name": null,
+  "trial_params": null
+}