Upload folder using huggingface_hub

README.md

@@ -1,3 +1,202 @@
----
-license: mit
----
+---
+base_model: nicholasKluge/TeenyTinyLlama-460m
+library_name: peft
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.15.2

adapter_config.json

@@ -0,0 +1,34 @@
+{
+  "alpha_pattern": {},
+  "auto_mapping": null,
+  "base_model_name_or_path": "nicholasKluge/TeenyTinyLlama-460m",
+  "bias": "none",
+  "corda_config": null,
+  "eva_config": null,
+  "exclude_modules": null,
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 16,
+  "lora_bias": false,
+  "lora_dropout": 0.05,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "r": 8,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "q_proj",
+    "v_proj"
+  ],
+  "task_type": "CAUSAL_LM",
+  "trainable_token_indices": null,
+  "use_dora": false,
+  "use_rslora": false
+}

adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:29bb1cf9d39944c9cd7115354bc81c47baf1cf2a25b9b9f24f8c1419e83b0e5e
+size 3158328

checkpoint-4100/README.md

@@ -0,0 +1,202 @@
+---
+base_model: nicholasKluge/TeenyTinyLlama-460m
+library_name: peft
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.15.2

checkpoint-4100/adapter_config.json

@@ -0,0 +1,34 @@
+{
+  "alpha_pattern": {},
+  "auto_mapping": null,
+  "base_model_name_or_path": "nicholasKluge/TeenyTinyLlama-460m",
+  "bias": "none",
+  "corda_config": null,
+  "eva_config": null,
+  "exclude_modules": null,
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 16,
+  "lora_bias": false,
+  "lora_dropout": 0.05,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "r": 8,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "q_proj",
+    "v_proj"
+  ],
+  "task_type": "CAUSAL_LM",
+  "trainable_token_indices": null,
+  "use_dora": false,
+  "use_rslora": false
+}

checkpoint-4100/adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5c282b0932b29fa8b31aa786af16ba0cd10d27d5804bae666827705c0b43b93f
+size 3158328

checkpoint-4100/optimizer.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:743fffc96dac65bc2bcd0e92d541672e259ac212868c6a9ecd7e1706bdc93e9e
+size 6372346

checkpoint-4100/rng_state.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:745713709acd4799b82fd9ea606d8eab39690fb80960e3173e95f7e77da8242c
+size 14244

checkpoint-4100/scaler.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ba088f0440b02886ae6e3eaa7b8fe5ba89ebc28957f09854b9bb0baf82ed4239
+size 988

checkpoint-4100/scheduler.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9499e14cd005b1233c1db1452c631e4509e84fd9e2d8e2376d9c7a05ce6e5cab
+size 1064

checkpoint-4100/special_tokens_map.json

@@ -0,0 +1,24 @@
+{
+  "bos_token": {
+    "content": "<s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "</s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": "</s>",
+  "unk_token": {
+    "content": "<unk>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

checkpoint-4100/tokenizer_config.json

@@ -0,0 +1,55 @@
+{
+  "add_bos_token": true,
+  "add_eos_token": false,
+  "add_prefix_space": null,
+  "added_tokens_decoder": {
+    "0": {
+      "content": "<unk>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "1": {
+      "content": "<s>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "2": {
+      "content": "</s>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "3": {
+      "content": "<pad>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "bos_token": "<s>",
+  "bos_token_id": 1,
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "</s>",
+  "eos_token_id": 2,
+  "extra_special_tokens": {},
+  "legacy": false,
+  "model_max_length": 1000000000000000019884624838656,
+  "pad_token": "</s>",
+  "pad_token_id": 2,
+  "padding_side": "right",
+  "sp_model_kwargs": {},
+  "tokenizer_class": "LlamaTokenizerFast",
+  "unk_token": "<unk>",
+  "unk_token_id": 0,
+  "use_default_system_prompt": false
+}

checkpoint-4100/trainer_state.json

@@ -0,0 +1,608 @@
+{
+  "best_global_step": null,
+  "best_metric": null,
+  "best_model_checkpoint": null,
+  "epoch": 3.9196940726577436,
+  "eval_steps": 500,
+  "global_step": 4100,
+  "is_hyper_param_search": false,
+  "is_local_process_zero": true,
+  "is_world_process_zero": true,
+  "log_history": [
+    {
+      "epoch": 0.04780114722753346,
+      "grad_norm": 0.0546465665102005,
+      "learning_rate": 4.941443594646272e-05,
+      "loss": 2.2922,
+      "step": 50
+    },
+    {
+      "epoch": 0.09560229445506692,
+      "grad_norm": 0.08099879324436188,
+      "learning_rate": 4.881692160611855e-05,
+      "loss": 2.2658,
+      "step": 100
+    },
+    {
+      "epoch": 0.14340344168260039,
+      "grad_norm": 0.08135072886943817,
+      "learning_rate": 4.821940726577438e-05,
+      "loss": 2.1959,
+      "step": 150
+    },
+    {
+      "epoch": 0.19120458891013384,
+      "grad_norm": 0.08765063434839249,
+      "learning_rate": 4.7621892925430214e-05,
+      "loss": 2.1285,
+      "step": 200
+    },
+    {
+      "epoch": 0.2390057361376673,
+      "grad_norm": 0.10307478904724121,
+      "learning_rate": 4.7024378585086046e-05,
+      "loss": 2.0899,
+      "step": 250
+    },
+    {
+      "epoch": 0.28680688336520077,
+      "grad_norm": 0.10948552936315536,
+      "learning_rate": 4.642686424474187e-05,
+      "loss": 2.0663,
+      "step": 300
+    },
+    {
+      "epoch": 0.33460803059273425,
+      "grad_norm": 0.1135614812374115,
+      "learning_rate": 4.5829349904397704e-05,
+      "loss": 2.0508,
+      "step": 350
+    },
+    {
+      "epoch": 0.3824091778202677,
+      "grad_norm": 0.12696176767349243,
+      "learning_rate": 4.5231835564053536e-05,
+      "loss": 1.9819,
+      "step": 400
+    },
+    {
+      "epoch": 0.43021032504780116,
+      "grad_norm": 0.13437969982624054,
+      "learning_rate": 4.463432122370937e-05,
+      "loss": 2.0004,
+      "step": 450
+    },
+    {
+      "epoch": 0.4780114722753346,
+      "grad_norm": 0.1275995969772339,
+      "learning_rate": 4.40368068833652e-05,
+      "loss": 1.9961,
+      "step": 500
+    },
+    {
+      "epoch": 0.5258126195028681,
+      "grad_norm": 0.14496305584907532,
+      "learning_rate": 4.343929254302104e-05,
+      "loss": 1.9855,
+      "step": 550
+    },
+    {
+      "epoch": 0.5736137667304015,
+      "grad_norm": 0.13401177525520325,
+      "learning_rate": 4.284177820267687e-05,
+      "loss": 1.976,
+      "step": 600
+    },
+    {
+      "epoch": 0.621414913957935,
+      "grad_norm": 0.15063685178756714,
+      "learning_rate": 4.2244263862332704e-05,
+      "loss": 1.9428,
+      "step": 650
+    },
+    {
+      "epoch": 0.6692160611854685,
+      "grad_norm": 0.1525646150112152,
+      "learning_rate": 4.164674952198853e-05,
+      "loss": 1.9565,
+      "step": 700
+    },
+    {
+      "epoch": 0.7170172084130019,
+      "grad_norm": 0.1553465574979782,
+      "learning_rate": 4.104923518164436e-05,
+      "loss": 1.9499,
+      "step": 750
+    },
+    {
+      "epoch": 0.7648183556405354,
+      "grad_norm": 0.178068146109581,
+      "learning_rate": 4.0451720841300193e-05,
+      "loss": 1.9342,
+      "step": 800
+    },
+    {
+      "epoch": 0.8126195028680688,
+      "grad_norm": 0.16074113547801971,
+      "learning_rate": 3.9854206500956026e-05,
+      "loss": 1.9304,
+      "step": 850
+    },
+    {
+      "epoch": 0.8604206500956023,
+      "grad_norm": 0.17012304067611694,
+      "learning_rate": 3.925669216061186e-05,
+      "loss": 1.9358,
+      "step": 900
+    },
+    {
+      "epoch": 0.9082217973231358,
+      "grad_norm": 0.1635637879371643,
+      "learning_rate": 3.865917782026769e-05,
+      "loss": 1.9145,
+      "step": 950
+    },
+    {
+      "epoch": 0.9560229445506692,
+      "grad_norm": 0.18287719786167145,
+      "learning_rate": 3.806166347992352e-05,
+      "loss": 1.9186,
+      "step": 1000
+    },
+    {
+      "epoch": 1.0038240917782026,
+      "grad_norm": 0.18034423887729645,
+      "learning_rate": 3.7464149139579354e-05,
+      "loss": 1.8995,
+      "step": 1050
+    },
+    {
+      "epoch": 1.0516252390057361,
+      "grad_norm": 0.16354230046272278,
+      "learning_rate": 3.6866634799235186e-05,
+      "loss": 1.9068,
+      "step": 1100
+    },
+    {
+      "epoch": 1.0994263862332696,
+      "grad_norm": 0.236989825963974,
+      "learning_rate": 3.626912045889101e-05,
+      "loss": 1.9141,
+      "step": 1150
+    },
+    {
+      "epoch": 1.147227533460803,
+      "grad_norm": 0.2356785088777542,
+      "learning_rate": 3.5671606118546844e-05,
+      "loss": 1.9088,
+      "step": 1200
+    },
+    {
+      "epoch": 1.1950286806883366,
+      "grad_norm": 0.1805247962474823,
+      "learning_rate": 3.5074091778202676e-05,
+      "loss": 1.8923,
+      "step": 1250
+    },
+    {
+      "epoch": 1.24282982791587,
+      "grad_norm": 0.21303710341453552,
+      "learning_rate": 3.447657743785851e-05,
+      "loss": 1.898,
+      "step": 1300
+    },
+    {
+      "epoch": 1.2906309751434035,
+      "grad_norm": 0.18716047704219818,
+      "learning_rate": 3.387906309751434e-05,
+      "loss": 1.8986,
+      "step": 1350
+    },
+    {
+      "epoch": 1.338432122370937,
+      "grad_norm": 0.16454172134399414,
+      "learning_rate": 3.328154875717017e-05,
+      "loss": 1.9149,
+      "step": 1400
+    },
+    {
+      "epoch": 1.3862332695984705,
+      "grad_norm": 0.18206411600112915,
+      "learning_rate": 3.2684034416826005e-05,
+      "loss": 1.8662,
+      "step": 1450
+    },
+    {
+      "epoch": 1.4340344168260037,
+      "grad_norm": 0.19803249835968018,
+      "learning_rate": 3.208652007648184e-05,
+      "loss": 1.9015,
+      "step": 1500
+    },
+    {
+      "epoch": 1.4818355640535372,
+      "grad_norm": 0.19260399043560028,
+      "learning_rate": 3.148900573613767e-05,
+      "loss": 1.905,
+      "step": 1550
+    },
+    {
+      "epoch": 1.5296367112810707,
+      "grad_norm": 0.20765070617198944,
+      "learning_rate": 3.08914913957935e-05,
+      "loss": 1.8981,
+      "step": 1600
+    },
+    {
+      "epoch": 1.5774378585086042,
+      "grad_norm": 0.1981675922870636,
+      "learning_rate": 3.029397705544933e-05,
+      "loss": 1.8957,
+      "step": 1650
+    },
+    {
+      "epoch": 1.6252390057361377,
+      "grad_norm": 0.19996145367622375,
+      "learning_rate": 2.9696462715105166e-05,
+      "loss": 1.8797,
+      "step": 1700
+    },
+    {
+      "epoch": 1.6730401529636711,
+      "grad_norm": 0.21547172963619232,
+      "learning_rate": 2.9098948374760998e-05,
+      "loss": 1.8919,
+      "step": 1750
+    },
+    {
+      "epoch": 1.7208413001912046,
+      "grad_norm": 0.21214577555656433,
+      "learning_rate": 2.850143403441683e-05,
+      "loss": 1.8942,
+      "step": 1800
+    },
+    {
+      "epoch": 1.7686424474187379,
+      "grad_norm": 0.19430747628211975,
+      "learning_rate": 2.7903919694072662e-05,
+      "loss": 1.8744,
+      "step": 1850
+    },
+    {
+      "epoch": 1.8164435946462714,
+      "grad_norm": 0.20672687888145447,
+      "learning_rate": 2.730640535372849e-05,
+      "loss": 1.8784,
+      "step": 1900
+    },
+    {
+      "epoch": 1.8642447418738048,
+      "grad_norm": 0.204985573887825,
+      "learning_rate": 2.6708891013384323e-05,
+      "loss": 1.8902,
+      "step": 1950
+    },
+    {
+      "epoch": 1.9120458891013383,
+      "grad_norm": 0.20450404286384583,
+      "learning_rate": 2.6111376673040155e-05,
+      "loss": 1.8649,
+      "step": 2000
+    },
+    {
+      "epoch": 1.9598470363288718,
+      "grad_norm": 0.20468485355377197,
+      "learning_rate": 2.5513862332695987e-05,
+      "loss": 1.8899,
+      "step": 2050
+    },
+    {
+      "epoch": 2.0076481835564053,
+      "grad_norm": 0.206816628575325,
+      "learning_rate": 2.491634799235182e-05,
+      "loss": 1.8607,
+      "step": 2100
+    },
+    {
+      "epoch": 2.0554493307839388,
+      "grad_norm": 0.2078491896390915,
+      "learning_rate": 2.431883365200765e-05,
+      "loss": 1.8737,
+      "step": 2150
+    },
+    {
+      "epoch": 2.1032504780114722,
+      "grad_norm": 0.19321344792842865,
+      "learning_rate": 2.372131931166348e-05,
+      "loss": 1.8597,
+      "step": 2200
+    },
+    {
+      "epoch": 2.1510516252390057,
+      "grad_norm": 0.22135953605175018,
+      "learning_rate": 2.3123804971319313e-05,
+      "loss": 1.88,
+      "step": 2250
+    },
+    {
+      "epoch": 2.198852772466539,
+      "grad_norm": 0.20404009521007538,
+      "learning_rate": 2.2526290630975145e-05,
+      "loss": 1.8908,
+      "step": 2300
+    },
+    {
+      "epoch": 2.2466539196940727,
+      "grad_norm": 0.21296437084674835,
+      "learning_rate": 2.1928776290630977e-05,
+      "loss": 1.8632,
+      "step": 2350
+    },
+    {
+      "epoch": 2.294455066921606,
+      "grad_norm": 0.2082318514585495,
+      "learning_rate": 2.1331261950286806e-05,
+      "loss": 1.8819,
+      "step": 2400
+    },
+    {
+      "epoch": 2.3422562141491396,
+      "grad_norm": 0.2119467705488205,
+      "learning_rate": 2.0733747609942638e-05,
+      "loss": 1.8679,
+      "step": 2450
+    },
+    {
+      "epoch": 2.390057361376673,
+      "grad_norm": 0.2162160575389862,
+      "learning_rate": 2.013623326959847e-05,
+      "loss": 1.8698,
+      "step": 2500
+    },
+    {
+      "epoch": 2.4378585086042066,
+      "grad_norm": 0.19639310240745544,
+      "learning_rate": 1.9538718929254306e-05,
+      "loss": 1.8808,
+      "step": 2550
+    },
+    {
+      "epoch": 2.48565965583174,
+      "grad_norm": 0.2149832397699356,
+      "learning_rate": 1.8941204588910135e-05,
+      "loss": 1.8743,
+      "step": 2600
+    },
+    {
+      "epoch": 2.5334608030592736,
+      "grad_norm": 0.20657892525196075,
+      "learning_rate": 1.8343690248565967e-05,
+      "loss": 1.8611,
+      "step": 2650
+    },
+    {
+      "epoch": 2.581261950286807,
+      "grad_norm": 0.22579824924468994,
+      "learning_rate": 1.77461759082218e-05,
+      "loss": 1.8834,
+      "step": 2700
+    },
+    {
+      "epoch": 2.62906309751434,
+      "grad_norm": 0.214285746216774,
+      "learning_rate": 1.714866156787763e-05,
+      "loss": 1.8712,
+      "step": 2750
+    },
+    {
+      "epoch": 2.676864244741874,
+      "grad_norm": 0.2056400179862976,
+      "learning_rate": 1.6551147227533463e-05,
+      "loss": 1.8515,
+      "step": 2800
+    },
+    {
+      "epoch": 2.724665391969407,
+      "grad_norm": 0.21018265187740326,
+      "learning_rate": 1.5953632887189292e-05,
+      "loss": 1.8537,
+      "step": 2850
+    },
+    {
+      "epoch": 2.772466539196941,
+      "grad_norm": 0.1982487589120865,
+      "learning_rate": 1.5356118546845124e-05,
+      "loss": 1.8637,
+      "step": 2900
+    },
+    {
+      "epoch": 2.820267686424474,
+      "grad_norm": 0.22870182991027832,
+      "learning_rate": 1.4758604206500956e-05,
+      "loss": 1.857,
+      "step": 2950
+    },
+    {
+      "epoch": 2.8680688336520075,
+      "grad_norm": 0.2098233997821808,
+      "learning_rate": 1.4161089866156787e-05,
+      "loss": 1.8433,
+      "step": 3000
+    },
+    {
+      "epoch": 2.915869980879541,
+      "grad_norm": 0.23259004950523376,
+      "learning_rate": 1.3563575525812619e-05,
+      "loss": 1.8934,
+      "step": 3050
+    },
+    {
+      "epoch": 2.9636711281070744,
+      "grad_norm": 0.2191799432039261,
+      "learning_rate": 1.2966061185468451e-05,
+      "loss": 1.8603,
+      "step": 3100
+    },
+    {
+      "epoch": 3.011472275334608,
+      "grad_norm": 0.21849119663238525,
+      "learning_rate": 1.2368546845124283e-05,
+      "loss": 1.8603,
+      "step": 3150
+    },
+    {
+      "epoch": 3.0592734225621414,
+      "grad_norm": 0.22140590846538544,
+      "learning_rate": 1.1771032504780115e-05,
+      "loss": 1.8696,
+      "step": 3200
+    },
+    {
+      "epoch": 3.107074569789675,
+      "grad_norm": 0.2533087432384491,
+      "learning_rate": 1.1173518164435948e-05,
+      "loss": 1.8491,
+      "step": 3250
+    },
+    {
+      "epoch": 3.1548757170172084,
+      "grad_norm": 0.22291821241378784,
+      "learning_rate": 1.057600382409178e-05,
+      "loss": 1.8664,
+      "step": 3300
+    },
+    {
+      "epoch": 3.202676864244742,
+      "grad_norm": 0.23389829695224762,
+      "learning_rate": 9.97848948374761e-06,
+      "loss": 1.8633,
+      "step": 3350
+    },
+    {
+      "epoch": 3.2504780114722753,
+      "grad_norm": 0.2284611165523529,
+      "learning_rate": 9.380975143403442e-06,
+      "loss": 1.8739,
+      "step": 3400
+    },
+    {
+      "epoch": 3.298279158699809,
+      "grad_norm": 0.2357141673564911,
+      "learning_rate": 8.783460803059273e-06,
+      "loss": 1.8837,
+      "step": 3450
+    },
+    {
+      "epoch": 3.3460803059273423,
+      "grad_norm": 0.23133474588394165,
+      "learning_rate": 8.185946462715105e-06,
+      "loss": 1.8284,
+      "step": 3500
+    },
+    {
+      "epoch": 3.3938814531548758,
+      "grad_norm": 0.24003466963768005,
+      "learning_rate": 7.588432122370938e-06,
+      "loss": 1.8557,
+      "step": 3550
+    },
+    {
+      "epoch": 3.4416826003824093,
+      "grad_norm": 0.22754019498825073,
+      "learning_rate": 6.990917782026769e-06,
+      "loss": 1.8726,
+      "step": 3600
+    },
+    {
+      "epoch": 3.4894837476099427,
+      "grad_norm": 0.24840685725212097,
+      "learning_rate": 6.393403441682601e-06,
+      "loss": 1.8683,
+      "step": 3650
+    },
+    {
+      "epoch": 3.537284894837476,
+      "grad_norm": 0.23282456398010254,
+      "learning_rate": 5.795889101338432e-06,
+      "loss": 1.8624,
+      "step": 3700
+    },
+    {
+      "epoch": 3.5850860420650097,
+      "grad_norm": 0.22449611127376556,
+      "learning_rate": 5.198374760994264e-06,
+      "loss": 1.8355,
+      "step": 3750
+    },
+    {
+      "epoch": 3.632887189292543,
+      "grad_norm": 0.22073350846767426,
+      "learning_rate": 4.6008604206500955e-06,
+      "loss": 1.8706,
+      "step": 3800
+    },
+    {
+      "epoch": 3.6806883365200767,
+      "grad_norm": 0.2504482567310333,
+      "learning_rate": 4.003346080305928e-06,
+      "loss": 1.8607,
+      "step": 3850
+    },
+    {
+      "epoch": 3.7284894837476097,
+      "grad_norm": 0.23835700750350952,
+      "learning_rate": 3.4058317399617594e-06,
+      "loss": 1.8495,
+      "step": 3900
+    },
+    {
+      "epoch": 3.7762906309751436,
+      "grad_norm": 0.2028919905424118,
+      "learning_rate": 2.808317399617591e-06,
+      "loss": 1.8423,
+      "step": 3950
+    },
+    {
+      "epoch": 3.8240917782026767,
+      "grad_norm": 0.23366276919841766,
+      "learning_rate": 2.210803059273423e-06,
+      "loss": 1.8431,
+      "step": 4000
+    },
+    {
+      "epoch": 3.8718929254302106,
+      "grad_norm": 0.261652410030365,
+      "learning_rate": 1.6132887189292542e-06,
+      "loss": 1.8639,
+      "step": 4050
+    },
+    {
+      "epoch": 3.9196940726577436,
+      "grad_norm": 0.2397291362285614,
+      "learning_rate": 1.015774378585086e-06,
+      "loss": 1.8626,
+      "step": 4100
+    }
+  ],
+  "logging_steps": 50,
+  "max_steps": 4184,
+  "num_input_tokens_seen": 0,
+  "num_train_epochs": 4,
+  "save_steps": 100,
+  "stateful_callbacks": {
+    "TrainerControl": {
+      "args": {
+        "should_epoch_stop": false,
+        "should_evaluate": false,
+        "should_log": false,
+        "should_save": true,
+        "should_training_stop": false
+      },
+      "attributes": {}
+    }
+  },
+  "total_flos": 5.491537700349542e+16,
+  "train_batch_size": 10,
+  "trial_name": null,
+  "trial_params": null
+}

checkpoint-4100/training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:bec7d7bd83c5b26e151b0650a8fa8ee73fdf56557e6c010410b27314702d49b2
+size 5304

checkpoint-4184/README.md

@@ -0,0 +1,202 @@
+---
+base_model: nicholasKluge/TeenyTinyLlama-460m
+library_name: peft
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.15.2

checkpoint-4184/adapter_config.json

@@ -0,0 +1,34 @@
+{
+  "alpha_pattern": {},
+  "auto_mapping": null,
+  "base_model_name_or_path": "nicholasKluge/TeenyTinyLlama-460m",
+  "bias": "none",
+  "corda_config": null,
+  "eva_config": null,
+  "exclude_modules": null,
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 16,
+  "lora_bias": false,
+  "lora_dropout": 0.05,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "r": 8,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "q_proj",
+    "v_proj"
+  ],
+  "task_type": "CAUSAL_LM",
+  "trainable_token_indices": null,
+  "use_dora": false,
+  "use_rslora": false
+}

checkpoint-4184/adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:29bb1cf9d39944c9cd7115354bc81c47baf1cf2a25b9b9f24f8c1419e83b0e5e
+size 3158328

checkpoint-4184/optimizer.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:bbd2782ae295435589c68f88b1ca31e9826f1e18a0e02f4753ca99bbc6d0f197
+size 6372346

checkpoint-4184/rng_state.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a2ea0d3bc88ebfaa8b26afbd8c644e33decfa843aa31af24e832b6faee63261d
+size 14244

checkpoint-4184/scaler.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5ecc38c7e74cd28fb3ef3b8ec21de9879422e35953ee9fac23cbbf13af21342e
+size 988

checkpoint-4184/scheduler.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:51ae83e6e5f81858e925ff02fdc609786471e9332dd3e08b34061492b3825938
+size 1064

checkpoint-4184/special_tokens_map.json

@@ -0,0 +1,24 @@
+{
+  "bos_token": {
+    "content": "<s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "</s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": "</s>",
+  "unk_token": {
+    "content": "<unk>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

checkpoint-4184/tokenizer_config.json

@@ -0,0 +1,55 @@
+{
+  "add_bos_token": true,
+  "add_eos_token": false,
+  "add_prefix_space": null,
+  "added_tokens_decoder": {
+    "0": {
+      "content": "<unk>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "1": {
+      "content": "<s>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "2": {
+      "content": "</s>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "3": {
+      "content": "<pad>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "bos_token": "<s>",
+  "bos_token_id": 1,
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "</s>",
+  "eos_token_id": 2,
+  "extra_special_tokens": {},
+  "legacy": false,
+  "model_max_length": 1000000000000000019884624838656,
+  "pad_token": "</s>",
+  "pad_token_id": 2,
+  "padding_side": "right",
+  "sp_model_kwargs": {},
+  "tokenizer_class": "LlamaTokenizerFast",
+  "unk_token": "<unk>",
+  "unk_token_id": 0,
+  "use_default_system_prompt": false
+}

checkpoint-4184/trainer_state.json

@@ -0,0 +1,615 @@
+{
+  "best_global_step": null,
+  "best_metric": null,
+  "best_model_checkpoint": null,
+  "epoch": 4.0,
+  "eval_steps": 500,
+  "global_step": 4184,
+  "is_hyper_param_search": false,
+  "is_local_process_zero": true,
+  "is_world_process_zero": true,
+  "log_history": [
+    {
+      "epoch": 0.04780114722753346,
+      "grad_norm": 0.0546465665102005,
+      "learning_rate": 4.941443594646272e-05,
+      "loss": 2.2922,
+      "step": 50
+    },
+    {
+      "epoch": 0.09560229445506692,
+      "grad_norm": 0.08099879324436188,
+      "learning_rate": 4.881692160611855e-05,
+      "loss": 2.2658,
+      "step": 100
+    },
+    {
+      "epoch": 0.14340344168260039,
+      "grad_norm": 0.08135072886943817,
+      "learning_rate": 4.821940726577438e-05,
+      "loss": 2.1959,
+      "step": 150
+    },
+    {
+      "epoch": 0.19120458891013384,
+      "grad_norm": 0.08765063434839249,
+      "learning_rate": 4.7621892925430214e-05,
+      "loss": 2.1285,
+      "step": 200
+    },
+    {
+      "epoch": 0.2390057361376673,
+      "grad_norm": 0.10307478904724121,
+      "learning_rate": 4.7024378585086046e-05,
+      "loss": 2.0899,
+      "step": 250
+    },
+    {
+      "epoch": 0.28680688336520077,
+      "grad_norm": 0.10948552936315536,
+      "learning_rate": 4.642686424474187e-05,
+      "loss": 2.0663,
+      "step": 300
+    },
+    {
+      "epoch": 0.33460803059273425,
+      "grad_norm": 0.1135614812374115,
+      "learning_rate": 4.5829349904397704e-05,
+      "loss": 2.0508,
+      "step": 350
+    },
+    {
+      "epoch": 0.3824091778202677,
+      "grad_norm": 0.12696176767349243,
+      "learning_rate": 4.5231835564053536e-05,
+      "loss": 1.9819,
+      "step": 400
+    },
+    {
+      "epoch": 0.43021032504780116,
+      "grad_norm": 0.13437969982624054,
+      "learning_rate": 4.463432122370937e-05,
+      "loss": 2.0004,
+      "step": 450
+    },
+    {
+      "epoch": 0.4780114722753346,
+      "grad_norm": 0.1275995969772339,
+      "learning_rate": 4.40368068833652e-05,
+      "loss": 1.9961,
+      "step": 500
+    },
+    {
+      "epoch": 0.5258126195028681,
+      "grad_norm": 0.14496305584907532,
+      "learning_rate": 4.343929254302104e-05,
+      "loss": 1.9855,
+      "step": 550
+    },
+    {
+      "epoch": 0.5736137667304015,
+      "grad_norm": 0.13401177525520325,
+      "learning_rate": 4.284177820267687e-05,
+      "loss": 1.976,
+      "step": 600
+    },
+    {
+      "epoch": 0.621414913957935,
+      "grad_norm": 0.15063685178756714,
+      "learning_rate": 4.2244263862332704e-05,
+      "loss": 1.9428,
+      "step": 650
+    },
+    {
+      "epoch": 0.6692160611854685,
+      "grad_norm": 0.1525646150112152,
+      "learning_rate": 4.164674952198853e-05,
+      "loss": 1.9565,
+      "step": 700
+    },
+    {
+      "epoch": 0.7170172084130019,
+      "grad_norm": 0.1553465574979782,
+      "learning_rate": 4.104923518164436e-05,
+      "loss": 1.9499,
+      "step": 750
+    },
+    {
+      "epoch": 0.7648183556405354,
+      "grad_norm": 0.178068146109581,
+      "learning_rate": 4.0451720841300193e-05,
+      "loss": 1.9342,
+      "step": 800
+    },
+    {
+      "epoch": 0.8126195028680688,
+      "grad_norm": 0.16074113547801971,
+      "learning_rate": 3.9854206500956026e-05,
+      "loss": 1.9304,
+      "step": 850
+    },
+    {
+      "epoch": 0.8604206500956023,
+      "grad_norm": 0.17012304067611694,
+      "learning_rate": 3.925669216061186e-05,
+      "loss": 1.9358,
+      "step": 900
+    },
+    {
+      "epoch": 0.9082217973231358,
+      "grad_norm": 0.1635637879371643,
+      "learning_rate": 3.865917782026769e-05,
+      "loss": 1.9145,
+      "step": 950
+    },
+    {
+      "epoch": 0.9560229445506692,
+      "grad_norm": 0.18287719786167145,
+      "learning_rate": 3.806166347992352e-05,
+      "loss": 1.9186,
+      "step": 1000
+    },
+    {
+      "epoch": 1.0038240917782026,
+      "grad_norm": 0.18034423887729645,
+      "learning_rate": 3.7464149139579354e-05,
+      "loss": 1.8995,
+      "step": 1050
+    },
+    {
+      "epoch": 1.0516252390057361,
+      "grad_norm": 0.16354230046272278,
+      "learning_rate": 3.6866634799235186e-05,
+      "loss": 1.9068,
+      "step": 1100
+    },
+    {
+      "epoch": 1.0994263862332696,
+      "grad_norm": 0.236989825963974,
+      "learning_rate": 3.626912045889101e-05,
+      "loss": 1.9141,
+      "step": 1150
+    },
+    {
+      "epoch": 1.147227533460803,
+      "grad_norm": 0.2356785088777542,
+      "learning_rate": 3.5671606118546844e-05,
+      "loss": 1.9088,
+      "step": 1200
+    },
+    {
+      "epoch": 1.1950286806883366,
+      "grad_norm": 0.1805247962474823,
+      "learning_rate": 3.5074091778202676e-05,
+      "loss": 1.8923,
+      "step": 1250
+    },
+    {
+      "epoch": 1.24282982791587,
+      "grad_norm": 0.21303710341453552,
+      "learning_rate": 3.447657743785851e-05,
+      "loss": 1.898,
+      "step": 1300
+    },
+    {
+      "epoch": 1.2906309751434035,
+      "grad_norm": 0.18716047704219818,
+      "learning_rate": 3.387906309751434e-05,
+      "loss": 1.8986,
+      "step": 1350
+    },
+    {
+      "epoch": 1.338432122370937,
+      "grad_norm": 0.16454172134399414,
+      "learning_rate": 3.328154875717017e-05,
+      "loss": 1.9149,
+      "step": 1400
+    },
+    {
+      "epoch": 1.3862332695984705,
+      "grad_norm": 0.18206411600112915,
+      "learning_rate": 3.2684034416826005e-05,
+      "loss": 1.8662,
+      "step": 1450
+    },
+    {
+      "epoch": 1.4340344168260037,
+      "grad_norm": 0.19803249835968018,
+      "learning_rate": 3.208652007648184e-05,
+      "loss": 1.9015,
+      "step": 1500
+    },
+    {
+      "epoch": 1.4818355640535372,
+      "grad_norm": 0.19260399043560028,
+      "learning_rate": 3.148900573613767e-05,
+      "loss": 1.905,
+      "step": 1550
+    },
+    {
+      "epoch": 1.5296367112810707,
+      "grad_norm": 0.20765070617198944,
+      "learning_rate": 3.08914913957935e-05,
+      "loss": 1.8981,
+      "step": 1600
+    },
+    {
+      "epoch": 1.5774378585086042,
+      "grad_norm": 0.1981675922870636,
+      "learning_rate": 3.029397705544933e-05,
+      "loss": 1.8957,
+      "step": 1650
+    },
+    {
+      "epoch": 1.6252390057361377,
+      "grad_norm": 0.19996145367622375,
+      "learning_rate": 2.9696462715105166e-05,
+      "loss": 1.8797,
+      "step": 1700
+    },
+    {
+      "epoch": 1.6730401529636711,
+      "grad_norm": 0.21547172963619232,
+      "learning_rate": 2.9098948374760998e-05,
+      "loss": 1.8919,
+      "step": 1750
+    },
+    {
+      "epoch": 1.7208413001912046,
+      "grad_norm": 0.21214577555656433,
+      "learning_rate": 2.850143403441683e-05,
+      "loss": 1.8942,
+      "step": 1800
+    },
+    {
+      "epoch": 1.7686424474187379,
+      "grad_norm": 0.19430747628211975,
+      "learning_rate": 2.7903919694072662e-05,
+      "loss": 1.8744,
+      "step": 1850
+    },
+    {
+      "epoch": 1.8164435946462714,
+      "grad_norm": 0.20672687888145447,
+      "learning_rate": 2.730640535372849e-05,
+      "loss": 1.8784,
+      "step": 1900
+    },
+    {
+      "epoch": 1.8642447418738048,
+      "grad_norm": 0.204985573887825,
+      "learning_rate": 2.6708891013384323e-05,
+      "loss": 1.8902,
+      "step": 1950
+    },
+    {
+      "epoch": 1.9120458891013383,
+      "grad_norm": 0.20450404286384583,
+      "learning_rate": 2.6111376673040155e-05,
+      "loss": 1.8649,
+      "step": 2000
+    },
+    {
+      "epoch": 1.9598470363288718,
+      "grad_norm": 0.20468485355377197,
+      "learning_rate": 2.5513862332695987e-05,
+      "loss": 1.8899,
+      "step": 2050
+    },
+    {
+      "epoch": 2.0076481835564053,
+      "grad_norm": 0.206816628575325,
+      "learning_rate": 2.491634799235182e-05,
+      "loss": 1.8607,
+      "step": 2100
+    },
+    {
+      "epoch": 2.0554493307839388,
+      "grad_norm": 0.2078491896390915,
+      "learning_rate": 2.431883365200765e-05,
+      "loss": 1.8737,
+      "step": 2150
+    },
+    {
+      "epoch": 2.1032504780114722,
+      "grad_norm": 0.19321344792842865,
+      "learning_rate": 2.372131931166348e-05,
+      "loss": 1.8597,
+      "step": 2200
+    },
+    {
+      "epoch": 2.1510516252390057,
+      "grad_norm": 0.22135953605175018,
+      "learning_rate": 2.3123804971319313e-05,
+      "loss": 1.88,
+      "step": 2250
+    },
+    {
+      "epoch": 2.198852772466539,
+      "grad_norm": 0.20404009521007538,
+      "learning_rate": 2.2526290630975145e-05,
+      "loss": 1.8908,
+      "step": 2300
+    },
+    {
+      "epoch": 2.2466539196940727,
+      "grad_norm": 0.21296437084674835,
+      "learning_rate": 2.1928776290630977e-05,
+      "loss": 1.8632,
+      "step": 2350
+    },
+    {
+      "epoch": 2.294455066921606,
+      "grad_norm": 0.2082318514585495,
+      "learning_rate": 2.1331261950286806e-05,
+      "loss": 1.8819,
+      "step": 2400
+    },
+    {
+      "epoch": 2.3422562141491396,
+      "grad_norm": 0.2119467705488205,
+      "learning_rate": 2.0733747609942638e-05,
+      "loss": 1.8679,
+      "step": 2450
+    },
+    {
+      "epoch": 2.390057361376673,
+      "grad_norm": 0.2162160575389862,
+      "learning_rate": 2.013623326959847e-05,
+      "loss": 1.8698,
+      "step": 2500
+    },
+    {
+      "epoch": 2.4378585086042066,
+      "grad_norm": 0.19639310240745544,
+      "learning_rate": 1.9538718929254306e-05,
+      "loss": 1.8808,
+      "step": 2550
+    },
+    {
+      "epoch": 2.48565965583174,
+      "grad_norm": 0.2149832397699356,
+      "learning_rate": 1.8941204588910135e-05,
+      "loss": 1.8743,
+      "step": 2600
+    },
+    {
+      "epoch": 2.5334608030592736,
+      "grad_norm": 0.20657892525196075,
+      "learning_rate": 1.8343690248565967e-05,
+      "loss": 1.8611,
+      "step": 2650
+    },
+    {
+      "epoch": 2.581261950286807,
+      "grad_norm": 0.22579824924468994,
+      "learning_rate": 1.77461759082218e-05,
+      "loss": 1.8834,
+      "step": 2700
+    },
+    {
+      "epoch": 2.62906309751434,
+      "grad_norm": 0.214285746216774,
+      "learning_rate": 1.714866156787763e-05,
+      "loss": 1.8712,
+      "step": 2750
+    },
+    {
+      "epoch": 2.676864244741874,
+      "grad_norm": 0.2056400179862976,
+      "learning_rate": 1.6551147227533463e-05,
+      "loss": 1.8515,
+      "step": 2800
+    },
+    {
+      "epoch": 2.724665391969407,
+      "grad_norm": 0.21018265187740326,
+      "learning_rate": 1.5953632887189292e-05,
+      "loss": 1.8537,
+      "step": 2850
+    },
+    {
+      "epoch": 2.772466539196941,
+      "grad_norm": 0.1982487589120865,
+      "learning_rate": 1.5356118546845124e-05,
+      "loss": 1.8637,
+      "step": 2900
+    },
+    {
+      "epoch": 2.820267686424474,
+      "grad_norm": 0.22870182991027832,
+      "learning_rate": 1.4758604206500956e-05,
+      "loss": 1.857,
+      "step": 2950
+    },
+    {
+      "epoch": 2.8680688336520075,
+      "grad_norm": 0.2098233997821808,
+      "learning_rate": 1.4161089866156787e-05,
+      "loss": 1.8433,
+      "step": 3000
+    },
+    {
+      "epoch": 2.915869980879541,
+      "grad_norm": 0.23259004950523376,
+      "learning_rate": 1.3563575525812619e-05,
+      "loss": 1.8934,
+      "step": 3050
+    },
+    {
+      "epoch": 2.9636711281070744,
+      "grad_norm": 0.2191799432039261,
+      "learning_rate": 1.2966061185468451e-05,
+      "loss": 1.8603,
+      "step": 3100
+    },
+    {
+      "epoch": 3.011472275334608,
+      "grad_norm": 0.21849119663238525,
+      "learning_rate": 1.2368546845124283e-05,
+      "loss": 1.8603,
+      "step": 3150
+    },
+    {
+      "epoch": 3.0592734225621414,
+      "grad_norm": 0.22140590846538544,
+      "learning_rate": 1.1771032504780115e-05,
+      "loss": 1.8696,
+      "step": 3200
+    },
+    {
+      "epoch": 3.107074569789675,
+      "grad_norm": 0.2533087432384491,
+      "learning_rate": 1.1173518164435948e-05,
+      "loss": 1.8491,
+      "step": 3250
+    },
+    {
+      "epoch": 3.1548757170172084,
+      "grad_norm": 0.22291821241378784,
+      "learning_rate": 1.057600382409178e-05,
+      "loss": 1.8664,
+      "step": 3300
+    },
+    {
+      "epoch": 3.202676864244742,
+      "grad_norm": 0.23389829695224762,
+      "learning_rate": 9.97848948374761e-06,
+      "loss": 1.8633,
+      "step": 3350
+    },
+    {
+      "epoch": 3.2504780114722753,
+      "grad_norm": 0.2284611165523529,
+      "learning_rate": 9.380975143403442e-06,
+      "loss": 1.8739,
+      "step": 3400
+    },
+    {
+      "epoch": 3.298279158699809,
+      "grad_norm": 0.2357141673564911,
+      "learning_rate": 8.783460803059273e-06,
+      "loss": 1.8837,
+      "step": 3450
+    },
+    {
+      "epoch": 3.3460803059273423,
+      "grad_norm": 0.23133474588394165,
+      "learning_rate": 8.185946462715105e-06,
+      "loss": 1.8284,
+      "step": 3500
+    },
+    {
+      "epoch": 3.3938814531548758,
+      "grad_norm": 0.24003466963768005,
+      "learning_rate": 7.588432122370938e-06,
+      "loss": 1.8557,
+      "step": 3550
+    },
+    {
+      "epoch": 3.4416826003824093,
+      "grad_norm": 0.22754019498825073,
+      "learning_rate": 6.990917782026769e-06,
+      "loss": 1.8726,
+      "step": 3600
+    },
+    {
+      "epoch": 3.4894837476099427,
+      "grad_norm": 0.24840685725212097,
+      "learning_rate": 6.393403441682601e-06,
+      "loss": 1.8683,
+      "step": 3650
+    },
+    {
+      "epoch": 3.537284894837476,
+      "grad_norm": 0.23282456398010254,
+      "learning_rate": 5.795889101338432e-06,
+      "loss": 1.8624,
+      "step": 3700
+    },
+    {
+      "epoch": 3.5850860420650097,
+      "grad_norm": 0.22449611127376556,
+      "learning_rate": 5.198374760994264e-06,
+      "loss": 1.8355,
+      "step": 3750
+    },
+    {
+      "epoch": 3.632887189292543,
+      "grad_norm": 0.22073350846767426,
+      "learning_rate": 4.6008604206500955e-06,
+      "loss": 1.8706,
+      "step": 3800
+    },
+    {
+      "epoch": 3.6806883365200767,
+      "grad_norm": 0.2504482567310333,
+      "learning_rate": 4.003346080305928e-06,
+      "loss": 1.8607,
+      "step": 3850
+    },
+    {
+      "epoch": 3.7284894837476097,
+      "grad_norm": 0.23835700750350952,
+      "learning_rate": 3.4058317399617594e-06,
+      "loss": 1.8495,
+      "step": 3900
+    },
+    {
+      "epoch": 3.7762906309751436,
+      "grad_norm": 0.2028919905424118,
+      "learning_rate": 2.808317399617591e-06,
+      "loss": 1.8423,
+      "step": 3950
+    },
+    {
+      "epoch": 3.8240917782026767,
+      "grad_norm": 0.23366276919841766,
+      "learning_rate": 2.210803059273423e-06,
+      "loss": 1.8431,
+      "step": 4000
+    },
+    {
+      "epoch": 3.8718929254302106,
+      "grad_norm": 0.261652410030365,
+      "learning_rate": 1.6132887189292542e-06,
+      "loss": 1.8639,
+      "step": 4050
+    },
+    {
+      "epoch": 3.9196940726577436,
+      "grad_norm": 0.2397291362285614,
+      "learning_rate": 1.015774378585086e-06,
+      "loss": 1.8626,
+      "step": 4100
+    },
+    {
+      "epoch": 3.967495219885277,
+      "grad_norm": 0.24784159660339355,
+      "learning_rate": 4.1826003824091784e-07,
+      "loss": 1.8817,
+      "step": 4150
+    }
+  ],
+  "logging_steps": 50,
+  "max_steps": 4184,
+  "num_input_tokens_seen": 0,
+  "num_train_epochs": 4,
+  "save_steps": 100,
+  "stateful_callbacks": {
+    "TrainerControl": {
+      "args": {
+        "should_epoch_stop": false,
+        "should_evaluate": false,
+        "should_log": false,
+        "should_save": true,
+        "should_training_stop": true
+      },
+      "attributes": {}
+    }
+  },
+  "total_flos": 5.603041003909939e+16,
+  "train_batch_size": 10,
+  "trial_name": null,
+  "trial_params": null
+}

checkpoint-4184/training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:bec7d7bd83c5b26e151b0650a8fa8ee73fdf56557e6c010410b27314702d49b2
+size 5304

special_tokens_map.json

@@ -0,0 +1,24 @@
+{
+  "bos_token": {
+    "content": "<s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "</s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": "</s>",
+  "unk_token": {
+    "content": "<unk>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

tokenizer_config.json

@@ -0,0 +1,55 @@
+{
+  "add_bos_token": true,
+  "add_eos_token": false,
+  "add_prefix_space": null,
+  "added_tokens_decoder": {
+    "0": {
+      "content": "<unk>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "1": {
+      "content": "<s>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "2": {
+      "content": "</s>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "3": {
+      "content": "<pad>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "bos_token": "<s>",
+  "bos_token_id": 1,
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "</s>",
+  "eos_token_id": 2,
+  "extra_special_tokens": {},
+  "legacy": false,
+  "model_max_length": 1000000000000000019884624838656,
+  "pad_token": "</s>",
+  "pad_token_id": 2,
+  "padding_side": "right",
+  "sp_model_kwargs": {},
+  "tokenizer_class": "LlamaTokenizerFast",
+  "unk_token": "<unk>",
+  "unk_token_id": 0,
+  "use_default_system_prompt": false
+}