Training in progress, step 100, checkpoint

last-checkpoint/README.md

@@ -0,0 +1,202 @@
+---
+base_model: furiosa-ai/mlperf-gpt-j-6b
+library_name: peft
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.13.2

last-checkpoint/adapter_config.json

@@ -0,0 +1,33 @@
+{
+  "alpha_pattern": {},
+  "auto_mapping": null,
+  "base_model_name_or_path": "furiosa-ai/mlperf-gpt-j-6b",
+  "bias": "none",
+  "fan_in_fan_out": null,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 128,
+  "lora_dropout": 0.1,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "r": 64,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "v_proj",
+    "fc_out",
+    "q_proj",
+    "fc_in",
+    "out_proj",
+    "k_proj"
+  ],
+  "task_type": "CAUSAL_LM",
+  "use_dora": false,
+  "use_rslora": false
+}

last-checkpoint/adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e88b70d2c835a58945d8ff955d1283178d447bbd381cf0f1f5549fdf579457c2
+size 528526760

last-checkpoint/added_tokens.json

@@ -0,0 +1,146 @@
+{
+  "<|extratoken_100|>": 50356,
+  "<|extratoken_101|>": 50357,
+  "<|extratoken_102|>": 50358,
+  "<|extratoken_103|>": 50359,
+  "<|extratoken_104|>": 50360,
+  "<|extratoken_105|>": 50361,
+  "<|extratoken_106|>": 50362,
+  "<|extratoken_107|>": 50363,
+  "<|extratoken_108|>": 50364,
+  "<|extratoken_109|>": 50365,
+  "<|extratoken_10|>": 50266,
+  "<|extratoken_110|>": 50366,
+  "<|extratoken_111|>": 50367,
+  "<|extratoken_112|>": 50368,
+  "<|extratoken_113|>": 50369,
+  "<|extratoken_114|>": 50370,
+  "<|extratoken_115|>": 50371,
+  "<|extratoken_116|>": 50372,
+  "<|extratoken_117|>": 50373,
+  "<|extratoken_118|>": 50374,
+  "<|extratoken_119|>": 50375,
+  "<|extratoken_11|>": 50267,
+  "<|extratoken_120|>": 50376,
+  "<|extratoken_121|>": 50377,
+  "<|extratoken_122|>": 50378,
+  "<|extratoken_123|>": 50379,
+  "<|extratoken_124|>": 50380,
+  "<|extratoken_125|>": 50381,
+  "<|extratoken_126|>": 50382,
+  "<|extratoken_127|>": 50383,
+  "<|extratoken_128|>": 50384,
+  "<|extratoken_129|>": 50385,
+  "<|extratoken_12|>": 50268,
+  "<|extratoken_130|>": 50386,
+  "<|extratoken_131|>": 50387,
+  "<|extratoken_132|>": 50388,
+  "<|extratoken_133|>": 50389,
+  "<|extratoken_134|>": 50390,
+  "<|extratoken_135|>": 50391,
+  "<|extratoken_136|>": 50392,
+  "<|extratoken_137|>": 50393,
+  "<|extratoken_138|>": 50394,
+  "<|extratoken_139|>": 50395,
+  "<|extratoken_13|>": 50269,
+  "<|extratoken_140|>": 50396,
+  "<|extratoken_141|>": 50397,
+  "<|extratoken_142|>": 50398,
+  "<|extratoken_143|>": 50399,
+  "<|extratoken_14|>": 50270,
+  "<|extratoken_15|>": 50271,
+  "<|extratoken_16|>": 50272,
+  "<|extratoken_17|>": 50273,
+  "<|extratoken_18|>": 50274,
+  "<|extratoken_19|>": 50275,
+  "<|extratoken_1|>": 50257,
+  "<|extratoken_20|>": 50276,
+  "<|extratoken_21|>": 50277,
+  "<|extratoken_22|>": 50278,
+  "<|extratoken_23|>": 50279,
+  "<|extratoken_24|>": 50280,
+  "<|extratoken_25|>": 50281,
+  "<|extratoken_26|>": 50282,
+  "<|extratoken_27|>": 50283,
+  "<|extratoken_28|>": 50284,
+  "<|extratoken_29|>": 50285,
+  "<|extratoken_2|>": 50258,
+  "<|extratoken_30|>": 50286,
+  "<|extratoken_31|>": 50287,
+  "<|extratoken_32|>": 50288,
+  "<|extratoken_33|>": 50289,
+  "<|extratoken_34|>": 50290,
+  "<|extratoken_35|>": 50291,
+  "<|extratoken_36|>": 50292,
+  "<|extratoken_37|>": 50293,
+  "<|extratoken_38|>": 50294,
+  "<|extratoken_39|>": 50295,
+  "<|extratoken_3|>": 50259,
+  "<|extratoken_40|>": 50296,
+  "<|extratoken_41|>": 50297,
+  "<|extratoken_42|>": 50298,
+  "<|extratoken_43|>": 50299,
+  "<|extratoken_44|>": 50300,
+  "<|extratoken_45|>": 50301,
+  "<|extratoken_46|>": 50302,
+  "<|extratoken_47|>": 50303,
+  "<|extratoken_48|>": 50304,
+  "<|extratoken_49|>": 50305,
+  "<|extratoken_4|>": 50260,
+  "<|extratoken_50|>": 50306,
+  "<|extratoken_51|>": 50307,
+  "<|extratoken_52|>": 50308,
+  "<|extratoken_53|>": 50309,
+  "<|extratoken_54|>": 50310,
+  "<|extratoken_55|>": 50311,
+  "<|extratoken_56|>": 50312,
+  "<|extratoken_57|>": 50313,
+  "<|extratoken_58|>": 50314,
+  "<|extratoken_59|>": 50315,
+  "<|extratoken_5|>": 50261,
+  "<|extratoken_60|>": 50316,
+  "<|extratoken_61|>": 50317,
+  "<|extratoken_62|>": 50318,
+  "<|extratoken_63|>": 50319,
+  "<|extratoken_64|>": 50320,
+  "<|extratoken_65|>": 50321,
+  "<|extratoken_66|>": 50322,
+  "<|extratoken_67|>": 50323,
+  "<|extratoken_68|>": 50324,
+  "<|extratoken_69|>": 50325,
+  "<|extratoken_6|>": 50262,
+  "<|extratoken_70|>": 50326,
+  "<|extratoken_71|>": 50327,
+  "<|extratoken_72|>": 50328,
+  "<|extratoken_73|>": 50329,
+  "<|extratoken_74|>": 50330,
+  "<|extratoken_75|>": 50331,
+  "<|extratoken_76|>": 50332,
+  "<|extratoken_77|>": 50333,
+  "<|extratoken_78|>": 50334,
+  "<|extratoken_79|>": 50335,
+  "<|extratoken_7|>": 50263,
+  "<|extratoken_80|>": 50336,
+  "<|extratoken_81|>": 50337,
+  "<|extratoken_82|>": 50338,
+  "<|extratoken_83|>": 50339,
+  "<|extratoken_84|>": 50340,
+  "<|extratoken_85|>": 50341,
+  "<|extratoken_86|>": 50342,
+  "<|extratoken_87|>": 50343,
+  "<|extratoken_88|>": 50344,
+  "<|extratoken_89|>": 50345,
+  "<|extratoken_8|>": 50264,
+  "<|extratoken_90|>": 50346,
+  "<|extratoken_91|>": 50347,
+  "<|extratoken_92|>": 50348,
+  "<|extratoken_93|>": 50349,
+  "<|extratoken_94|>": 50350,
+  "<|extratoken_95|>": 50351,
+  "<|extratoken_96|>": 50352,
+  "<|extratoken_97|>": 50353,
+  "<|extratoken_98|>": 50354,
+  "<|extratoken_99|>": 50355,
+  "<|extratoken_9|>": 50265,
+  "[PAD]": 50400
+}

last-checkpoint/optimizer.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:bc680880a070fa2af4845e1c0ca748260113a92e8354eaad57575998662e0408
+size 141172038

last-checkpoint/rng_state.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d7a796eca2eedeff2f86feb608bdf1ec2be80e0e0041219e6b589d7624b3c0cf
+size 14244

last-checkpoint/scheduler.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9422835f6e08cdee44ef7e345a9055a0697e164408307b543cc423c63098f93c
+size 2080

last-checkpoint/special_tokens_map.json

@@ -0,0 +1,30 @@
+{
+  "bos_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  },
+  "unk_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  }
+}

last-checkpoint/tokenizer_config.json

@@ -0,0 +1,1176 @@
+{
+  "add_bos_token": false,
+  "add_prefix_space": false,
+  "added_tokens_decoder": {
+    "50256": {
+      "content": "<|endoftext|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "50257": {
+      "content": "<|extratoken_1|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50258": {
+      "content": "<|extratoken_2|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50259": {
+      "content": "<|extratoken_3|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50260": {
+      "content": "<|extratoken_4|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50261": {
+      "content": "<|extratoken_5|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50262": {
+      "content": "<|extratoken_6|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50263": {
+      "content": "<|extratoken_7|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50264": {
+      "content": "<|extratoken_8|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50265": {
+      "content": "<|extratoken_9|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50266": {
+      "content": "<|extratoken_10|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50267": {
+      "content": "<|extratoken_11|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50268": {
+      "content": "<|extratoken_12|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50269": {
+      "content": "<|extratoken_13|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50270": {
+      "content": "<|extratoken_14|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50271": {
+      "content": "<|extratoken_15|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50272": {
+      "content": "<|extratoken_16|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50273": {
+      "content": "<|extratoken_17|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50274": {
+      "content": "<|extratoken_18|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50275": {
+      "content": "<|extratoken_19|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50276": {
+      "content": "<|extratoken_20|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50277": {
+      "content": "<|extratoken_21|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50278": {
+      "content": "<|extratoken_22|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50279": {
+      "content": "<|extratoken_23|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50280": {
+      "content": "<|extratoken_24|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50281": {
+      "content": "<|extratoken_25|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50282": {
+      "content": "<|extratoken_26|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50283": {
+      "content": "<|extratoken_27|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50284": {
+      "content": "<|extratoken_28|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50285": {
+      "content": "<|extratoken_29|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50286": {
+      "content": "<|extratoken_30|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50287": {
+      "content": "<|extratoken_31|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50288": {
+      "content": "<|extratoken_32|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50289": {
+      "content": "<|extratoken_33|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50290": {
+      "content": "<|extratoken_34|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50291": {
+      "content": "<|extratoken_35|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50292": {
+      "content": "<|extratoken_36|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50293": {
+      "content": "<|extratoken_37|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50294": {
+      "content": "<|extratoken_38|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50295": {
+      "content": "<|extratoken_39|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50296": {
+      "content": "<|extratoken_40|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50297": {
+      "content": "<|extratoken_41|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50298": {
+      "content": "<|extratoken_42|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50299": {
+      "content": "<|extratoken_43|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50300": {
+      "content": "<|extratoken_44|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50301": {
+      "content": "<|extratoken_45|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50302": {
+      "content": "<|extratoken_46|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50303": {
+      "content": "<|extratoken_47|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50304": {
+      "content": "<|extratoken_48|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50305": {
+      "content": "<|extratoken_49|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50306": {
+      "content": "<|extratoken_50|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50307": {
+      "content": "<|extratoken_51|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50308": {
+      "content": "<|extratoken_52|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50309": {
+      "content": "<|extratoken_53|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50310": {
+      "content": "<|extratoken_54|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50311": {
+      "content": "<|extratoken_55|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50312": {
+      "content": "<|extratoken_56|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50313": {
+      "content": "<|extratoken_57|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50314": {
+      "content": "<|extratoken_58|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50315": {
+      "content": "<|extratoken_59|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50316": {
+      "content": "<|extratoken_60|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50317": {
+      "content": "<|extratoken_61|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50318": {
+      "content": "<|extratoken_62|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50319": {
+      "content": "<|extratoken_63|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50320": {
+      "content": "<|extratoken_64|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50321": {
+      "content": "<|extratoken_65|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50322": {
+      "content": "<|extratoken_66|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50323": {
+      "content": "<|extratoken_67|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50324": {
+      "content": "<|extratoken_68|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50325": {
+      "content": "<|extratoken_69|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50326": {
+      "content": "<|extratoken_70|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50327": {
+      "content": "<|extratoken_71|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50328": {
+      "content": "<|extratoken_72|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50329": {
+      "content": "<|extratoken_73|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50330": {
+      "content": "<|extratoken_74|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50331": {
+      "content": "<|extratoken_75|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50332": {
+      "content": "<|extratoken_76|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50333": {
+      "content": "<|extratoken_77|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50334": {
+      "content": "<|extratoken_78|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50335": {
+      "content": "<|extratoken_79|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50336": {
+      "content": "<|extratoken_80|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50337": {
+      "content": "<|extratoken_81|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50338": {
+      "content": "<|extratoken_82|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50339": {
+      "content": "<|extratoken_83|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50340": {
+      "content": "<|extratoken_84|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50341": {
+      "content": "<|extratoken_85|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50342": {
+      "content": "<|extratoken_86|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50343": {
+      "content": "<|extratoken_87|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50344": {
+      "content": "<|extratoken_88|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50345": {
+      "content": "<|extratoken_89|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50346": {
+      "content": "<|extratoken_90|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50347": {
+      "content": "<|extratoken_91|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50348": {
+      "content": "<|extratoken_92|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50349": {
+      "content": "<|extratoken_93|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50350": {
+      "content": "<|extratoken_94|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50351": {
+      "content": "<|extratoken_95|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50352": {
+      "content": "<|extratoken_96|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50353": {
+      "content": "<|extratoken_97|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50354": {
+      "content": "<|extratoken_98|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50355": {
+      "content": "<|extratoken_99|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50356": {
+      "content": "<|extratoken_100|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50357": {
+      "content": "<|extratoken_101|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50358": {
+      "content": "<|extratoken_102|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50359": {
+      "content": "<|extratoken_103|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50360": {
+      "content": "<|extratoken_104|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50361": {
+      "content": "<|extratoken_105|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50362": {
+      "content": "<|extratoken_106|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50363": {
+      "content": "<|extratoken_107|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50364": {
+      "content": "<|extratoken_108|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50365": {
+      "content": "<|extratoken_109|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50366": {
+      "content": "<|extratoken_110|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50367": {
+      "content": "<|extratoken_111|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50368": {
+      "content": "<|extratoken_112|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50369": {
+      "content": "<|extratoken_113|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50370": {
+      "content": "<|extratoken_114|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50371": {
+      "content": "<|extratoken_115|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50372": {
+      "content": "<|extratoken_116|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50373": {
+      "content": "<|extratoken_117|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50374": {
+      "content": "<|extratoken_118|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50375": {
+      "content": "<|extratoken_119|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50376": {
+      "content": "<|extratoken_120|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50377": {
+      "content": "<|extratoken_121|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50378": {
+      "content": "<|extratoken_122|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50379": {
+      "content": "<|extratoken_123|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50380": {
+      "content": "<|extratoken_124|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50381": {
+      "content": "<|extratoken_125|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50382": {
+      "content": "<|extratoken_126|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50383": {
+      "content": "<|extratoken_127|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50384": {
+      "content": "<|extratoken_128|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50385": {
+      "content": "<|extratoken_129|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50386": {
+      "content": "<|extratoken_130|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50387": {
+      "content": "<|extratoken_131|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50388": {
+      "content": "<|extratoken_132|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50389": {
+      "content": "<|extratoken_133|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50390": {
+      "content": "<|extratoken_134|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50391": {
+      "content": "<|extratoken_135|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50392": {
+      "content": "<|extratoken_136|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50393": {
+      "content": "<|extratoken_137|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50394": {
+      "content": "<|extratoken_138|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50395": {
+      "content": "<|extratoken_139|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50396": {
+      "content": "<|extratoken_140|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50397": {
+      "content": "<|extratoken_141|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50398": {
+      "content": "<|extratoken_142|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50399": {
+      "content": "<|extratoken_143|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50400": {
+      "content": "[PAD]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    }
+  },
+  "bos_token": "<|endoftext|>",
+  "chat_template": "{% if not add_generation_prompt is defined %}{% set add_generation_prompt = false %}{% endif %}{% set loop_messages = messages %}{% for message in loop_messages %}{% set content = '<|start_header_id|>' + message['role'] + '<|end_header_id|>\n\n'+ message['content'] | trim + '<|eot_id|>' %}{% if loop.index0 == 0 %}{% set content = bos_token + content %}{% endif %}{{ content }}{% endfor %}{% if add_generation_prompt %}{{ '<|start_header_id|>assistant<|end_header_id|>\n\n' }}{% endif %}",
+  "clean_up_tokenization_spaces": true,
+  "eos_token": "<|endoftext|>",
+  "errors": "replace",
+  "model_max_length": 1919,
+  "pad_token": "<|endoftext|>",
+  "padding_side": "left",
+  "tokenizer_class": "GPT2Tokenizer",
+  "unk_token": "<|endoftext|>"
+}

last-checkpoint/trainer_state.json

@@ -0,0 +1,758 @@
+{
+  "best_metric": 0.864631175994873,
+  "best_model_checkpoint": "miner_id_24/checkpoint-100",
+  "epoch": 0.01876700760063808,
+  "eval_steps": 100,
+  "global_step": 100,
+  "is_hyper_param_search": false,
+  "is_local_process_zero": true,
+  "is_world_process_zero": true,
+  "log_history": [
+    {
+      "epoch": 0.0001876700760063808,
+      "grad_norm": 12.81955337524414,
+      "learning_rate": 6.666666649834951e-06,
+      "loss": 12.8442,
+      "step": 1
+    },
+    {
+      "epoch": 0.0001876700760063808,
+      "eval_loss": 3.0495126247406006,
+      "eval_runtime": 60.8504,
+      "eval_samples_per_second": 3.533,
+      "eval_steps_per_second": 1.775,
+      "step": 1
+    },
+    {
+      "epoch": 0.0003753401520127616,
+      "grad_norm": 13.319238662719727,
+      "learning_rate": 1.3333333299669903e-05,
+      "loss": 12.4649,
+      "step": 2
+    },
+    {
+      "epoch": 0.0005630102280191424,
+      "grad_norm": 8.796210289001465,
+      "learning_rate": 1.9999999494757503e-05,
+      "loss": 9.5825,
+      "step": 3
+    },
+    {
+      "epoch": 0.0007506803040255232,
+      "grad_norm": 13.615736961364746,
+      "learning_rate": 2.6666666599339806e-05,
+      "loss": 12.044,
+      "step": 4
+    },
+    {
+      "epoch": 0.000938350380031904,
+      "grad_norm": 10.25422191619873,
+      "learning_rate": 3.333333370392211e-05,
+      "loss": 11.3815,
+      "step": 5
+    },
+    {
+      "epoch": 0.0011260204560382848,
+      "grad_norm": 8.191003799438477,
+      "learning_rate": 3.9999998989515007e-05,
+      "loss": 9.6081,
+      "step": 6
+    },
+    {
+      "epoch": 0.0013136905320446654,
+      "grad_norm": 9.084091186523438,
+      "learning_rate": 4.6666664275107905e-05,
+      "loss": 9.1392,
+      "step": 7
+    },
+    {
+      "epoch": 0.0015013606080510463,
+      "grad_norm": 10.759261131286621,
+      "learning_rate": 5.333333319867961e-05,
+      "loss": 9.7277,
+      "step": 8
+    },
+    {
+      "epoch": 0.001689030684057427,
+      "grad_norm": 13.943377494812012,
+      "learning_rate": 6.000000212225132e-05,
+      "loss": 11.1142,
+      "step": 9
+    },
+    {
+      "epoch": 0.001876700760063808,
+      "grad_norm": 9.979884147644043,
+      "learning_rate": 6.666666740784422e-05,
+      "loss": 8.009,
+      "step": 10
+    },
+    {
+      "epoch": 0.0020643708360701886,
+      "grad_norm": 11.828911781311035,
+      "learning_rate": 7.333333633141592e-05,
+      "loss": 9.2504,
+      "step": 11
+    },
+    {
+      "epoch": 0.0022520409120765695,
+      "grad_norm": 7.584584712982178,
+      "learning_rate": 7.999999797903001e-05,
+      "loss": 6.8016,
+      "step": 12
+    },
+    {
+      "epoch": 0.00243971098808295,
+      "grad_norm": 9.295977592468262,
+      "learning_rate": 8.666666690260172e-05,
+      "loss": 7.1333,
+      "step": 13
+    },
+    {
+      "epoch": 0.002627381064089331,
+      "grad_norm": 11.4473876953125,
+      "learning_rate": 9.333332855021581e-05,
+      "loss": 6.8021,
+      "step": 14
+    },
+    {
+      "epoch": 0.0028150511400957118,
+      "grad_norm": 9.786142349243164,
+      "learning_rate": 9.999999747378752e-05,
+      "loss": 6.7248,
+      "step": 15
+    },
+    {
+      "epoch": 0.0030027212161020927,
+      "grad_norm": 11.391505241394043,
+      "learning_rate": 0.00010666666639735922,
+      "loss": 6.7056,
+      "step": 16
+    },
+    {
+      "epoch": 0.003190391292108473,
+      "grad_norm": 10.189046859741211,
+      "learning_rate": 0.00011333332804497331,
+      "loss": 4.7453,
+      "step": 17
+    },
+    {
+      "epoch": 0.003378061368114854,
+      "grad_norm": 16.046733856201172,
+      "learning_rate": 0.00012000000424450263,
+      "loss": 6.6072,
+      "step": 18
+    },
+    {
+      "epoch": 0.003565731444121235,
+      "grad_norm": 12.03208065032959,
+      "learning_rate": 0.00012666666589211673,
+      "loss": 5.9597,
+      "step": 19
+    },
+    {
+      "epoch": 0.003753401520127616,
+      "grad_norm": 8.420626640319824,
+      "learning_rate": 0.00013333333481568843,
+      "loss": 5.4908,
+      "step": 20
+    },
+    {
+      "epoch": 0.003941071596133996,
+      "grad_norm": 10.598747253417969,
+      "learning_rate": 0.0001399999891873449,
+      "loss": 5.1553,
+      "step": 21
+    },
+    {
+      "epoch": 0.004128741672140377,
+      "grad_norm": 11.755189895629883,
+      "learning_rate": 0.00014666667266283184,
+      "loss": 5.984,
+      "step": 22
+    },
+    {
+      "epoch": 0.004316411748146758,
+      "grad_norm": 9.006994247436523,
+      "learning_rate": 0.00015333332703448832,
+      "loss": 5.8985,
+      "step": 23
+    },
+    {
+      "epoch": 0.004504081824153139,
+      "grad_norm": 6.949065208435059,
+      "learning_rate": 0.00015999999595806003,
+      "loss": 4.7649,
+      "step": 24
+    },
+    {
+      "epoch": 0.00469175190015952,
+      "grad_norm": 8.948777198791504,
+      "learning_rate": 0.00016666666488163173,
+      "loss": 6.1666,
+      "step": 25
+    },
+    {
+      "epoch": 0.0048794219761659,
+      "grad_norm": 6.198967456817627,
+      "learning_rate": 0.00017333333380520344,
+      "loss": 4.0053,
+      "step": 26
+    },
+    {
+      "epoch": 0.005067092052172281,
+      "grad_norm": 5.79871129989624,
+      "learning_rate": 0.00017999998817685992,
+      "loss": 2.6955,
+      "step": 27
+    },
+    {
+      "epoch": 0.005254762128178662,
+      "grad_norm": 9.885784149169922,
+      "learning_rate": 0.00018666665710043162,
+      "loss": 5.525,
+      "step": 28
+    },
+    {
+      "epoch": 0.005442432204185043,
+      "grad_norm": 6.712244987487793,
+      "learning_rate": 0.00019333332602400333,
+      "loss": 2.9028,
+      "step": 29
+    },
+    {
+      "epoch": 0.0056301022801914236,
+      "grad_norm": 6.6082682609558105,
+      "learning_rate": 0.00019999999494757503,
+      "loss": 5.033,
+      "step": 30
+    },
+    {
+      "epoch": 0.0058177723561978045,
+      "grad_norm": 5.452151775360107,
+      "learning_rate": 0.00019999999494757503,
+      "loss": 3.3493,
+      "step": 31
+    },
+    {
+      "epoch": 0.006005442432204185,
+      "grad_norm": 10.815617561340332,
+      "learning_rate": 0.0001999999803956598,
+      "loss": 4.9692,
+      "step": 32
+    },
+    {
+      "epoch": 0.006193112508210565,
+      "grad_norm": 5.885110855102539,
+      "learning_rate": 0.0001999999803956598,
+      "loss": 4.9723,
+      "step": 33
+    },
+    {
+      "epoch": 0.006380782584216946,
+      "grad_norm": 7.814040660858154,
+      "learning_rate": 0.00019999996584374458,
+      "loss": 4.1859,
+      "step": 34
+    },
+    {
+      "epoch": 0.006568452660223327,
+      "grad_norm": 9.663604736328125,
+      "learning_rate": 0.00019999995129182935,
+      "loss": 4.7371,
+      "step": 35
+    },
+    {
+      "epoch": 0.006756122736229708,
+      "grad_norm": 4.464984893798828,
+      "learning_rate": 0.0001999999221879989,
+      "loss": 4.0791,
+      "step": 36
+    },
+    {
+      "epoch": 0.006943792812236089,
+      "grad_norm": 8.492260932922363,
+      "learning_rate": 0.00019999989308416843,
+      "loss": 5.1274,
+      "step": 37
+    },
+    {
+      "epoch": 0.00713146288824247,
+      "grad_norm": 8.78378963470459,
+      "learning_rate": 0.0001999998785322532,
+      "loss": 4.3128,
+      "step": 38
+    },
+    {
+      "epoch": 0.007319132964248851,
+      "grad_norm": 11.416115760803223,
+      "learning_rate": 0.00019999983487650752,
+      "loss": 3.3235,
+      "step": 39
+    },
+    {
+      "epoch": 0.007506803040255232,
+      "grad_norm": 6.054617881774902,
+      "learning_rate": 0.00019999980577267706,
+      "loss": 4.745,
+      "step": 40
+    },
+    {
+      "epoch": 0.007694473116261612,
+      "grad_norm": 5.310433864593506,
+      "learning_rate": 0.00019999976211693138,
+      "loss": 3.753,
+      "step": 41
+    },
+    {
+      "epoch": 0.007882143192267993,
+      "grad_norm": 12.702404022216797,
+      "learning_rate": 0.0001999997184611857,
+      "loss": 5.5771,
+      "step": 42
+    },
+    {
+      "epoch": 0.008069813268274374,
+      "grad_norm": 4.548901081085205,
+      "learning_rate": 0.00019999967480544,
+      "loss": 2.6456,
+      "step": 43
+    },
+    {
+      "epoch": 0.008257483344280754,
+      "grad_norm": 7.563997268676758,
+      "learning_rate": 0.0001999996165977791,
+      "loss": 3.9797,
+      "step": 44
+    },
+    {
+      "epoch": 0.008445153420287135,
+      "grad_norm": 6.839761734008789,
+      "learning_rate": 0.00019999955839011818,
+      "loss": 3.2799,
+      "step": 45
+    },
+    {
+      "epoch": 0.008632823496293516,
+      "grad_norm": 7.820802688598633,
+      "learning_rate": 0.00019999950018245727,
+      "loss": 4.035,
+      "step": 46
+    },
+    {
+      "epoch": 0.008820493572299897,
+      "grad_norm": 14.55642318725586,
+      "learning_rate": 0.00019999942742288113,
+      "loss": 6.6003,
+      "step": 47
+    },
+    {
+      "epoch": 0.009008163648306278,
+      "grad_norm": 5.329503536224365,
+      "learning_rate": 0.0001999993692152202,
+      "loss": 3.845,
+      "step": 48
+    },
+    {
+      "epoch": 0.009195833724312659,
+      "grad_norm": 6.151081562042236,
+      "learning_rate": 0.00019999929645564407,
+      "loss": 2.7,
+      "step": 49
+    },
+    {
+      "epoch": 0.00938350380031904,
+      "grad_norm": 6.816937446594238,
+      "learning_rate": 0.00019999922369606793,
+      "loss": 4.3751,
+      "step": 50
+    },
+    {
+      "epoch": 0.00957117387632542,
+      "grad_norm": 9.370729446411133,
+      "learning_rate": 0.00019999913638457656,
+      "loss": 4.8368,
+      "step": 51
+    },
+    {
+      "epoch": 0.0097588439523318,
+      "grad_norm": 5.665166854858398,
+      "learning_rate": 0.0001999990490730852,
+      "loss": 3.7484,
+      "step": 52
+    },
+    {
+      "epoch": 0.00994651402833818,
+      "grad_norm": 5.613337516784668,
+      "learning_rate": 0.00019999897631350905,
+      "loss": 3.0527,
+      "step": 53
+    },
+    {
+      "epoch": 0.010134184104344562,
+      "grad_norm": 5.777795791625977,
+      "learning_rate": 0.00019999887445010245,
+      "loss": 2.4735,
+      "step": 54
+    },
+    {
+      "epoch": 0.010321854180350943,
+      "grad_norm": 5.276432037353516,
+      "learning_rate": 0.00019999877258669585,
+      "loss": 3.3067,
+      "step": 55
+    },
+    {
+      "epoch": 0.010509524256357324,
+      "grad_norm": 8.136743545532227,
+      "learning_rate": 0.00019999868527520448,
+      "loss": 4.7536,
+      "step": 56
+    },
+    {
+      "epoch": 0.010697194332363704,
+      "grad_norm": 4.0192694664001465,
+      "learning_rate": 0.00019999858341179788,
+      "loss": 2.4309,
+      "step": 57
+    },
+    {
+      "epoch": 0.010884864408370085,
+      "grad_norm": 6.856616020202637,
+      "learning_rate": 0.00019999846699647605,
+      "loss": 4.3331,
+      "step": 58
+    },
+    {
+      "epoch": 0.011072534484376466,
+      "grad_norm": 5.82205057144165,
+      "learning_rate": 0.00019999836513306946,
+      "loss": 2.4353,
+      "step": 59
+    },
+    {
+      "epoch": 0.011260204560382847,
+      "grad_norm": 5.513638496398926,
+      "learning_rate": 0.00019999824871774763,
+      "loss": 3.339,
+      "step": 60
+    },
+    {
+      "epoch": 0.011447874636389228,
+      "grad_norm": 5.644273281097412,
+      "learning_rate": 0.0001999981323024258,
+      "loss": 2.6956,
+      "step": 61
+    },
+    {
+      "epoch": 0.011635544712395609,
+      "grad_norm": 5.618515491485596,
+      "learning_rate": 0.00019999800133518875,
+      "loss": 3.684,
+      "step": 62
+    },
+    {
+      "epoch": 0.01182321478840199,
+      "grad_norm": 6.951975345611572,
+      "learning_rate": 0.00019999788491986692,
+      "loss": 4.2561,
+      "step": 63
+    },
+    {
+      "epoch": 0.01201088486440837,
+      "grad_norm": 6.014930248260498,
+      "learning_rate": 0.00019999775395262986,
+      "loss": 3.3327,
+      "step": 64
+    },
+    {
+      "epoch": 0.012198554940414752,
+      "grad_norm": 4.603240966796875,
+      "learning_rate": 0.0001999976229853928,
+      "loss": 3.6254,
+      "step": 65
+    },
+    {
+      "epoch": 0.01238622501642113,
+      "grad_norm": 4.194943904876709,
+      "learning_rate": 0.00019999747746624053,
+      "loss": 3.1691,
+      "step": 66
+    },
+    {
+      "epoch": 0.012573895092427512,
+      "grad_norm": 7.527845859527588,
+      "learning_rate": 0.00019999733194708824,
+      "loss": 3.7516,
+      "step": 67
+    },
+    {
+      "epoch": 0.012761565168433893,
+      "grad_norm": 7.276858806610107,
+      "learning_rate": 0.00019999718642793596,
+      "loss": 4.0773,
+      "step": 68
+    },
+    {
+      "epoch": 0.012949235244440273,
+      "grad_norm": 7.401817798614502,
+      "learning_rate": 0.0001999970554606989,
+      "loss": 3.0396,
+      "step": 69
+    },
+    {
+      "epoch": 0.013136905320446654,
+      "grad_norm": 4.466447353363037,
+      "learning_rate": 0.0001999968953896314,
+      "loss": 3.4235,
+      "step": 70
+    },
+    {
+      "epoch": 0.013324575396453035,
+      "grad_norm": 10.029444694519043,
+      "learning_rate": 0.00019999673531856388,
+      "loss": 5.1296,
+      "step": 71
+    },
+    {
+      "epoch": 0.013512245472459416,
+      "grad_norm": 6.834356784820557,
+      "learning_rate": 0.00019999657524749637,
+      "loss": 3.5959,
+      "step": 72
+    },
+    {
+      "epoch": 0.013699915548465797,
+      "grad_norm": 3.7514798641204834,
+      "learning_rate": 0.00019999640062451363,
+      "loss": 2.6647,
+      "step": 73
+    },
+    {
+      "epoch": 0.013887585624472178,
+      "grad_norm": 7.09303092956543,
+      "learning_rate": 0.00019999624055344611,
+      "loss": 3.3474,
+      "step": 74
+    },
+    {
+      "epoch": 0.014075255700478559,
+      "grad_norm": 5.362713813781738,
+      "learning_rate": 0.00019999606593046337,
+      "loss": 3.3301,
+      "step": 75
+    },
+    {
+      "epoch": 0.01426292577648494,
+      "grad_norm": 5.224695682525635,
+      "learning_rate": 0.00019999589130748063,
+      "loss": 2.8853,
+      "step": 76
+    },
+    {
+      "epoch": 0.01445059585249132,
+      "grad_norm": 6.087166786193848,
+      "learning_rate": 0.0001999957166844979,
+      "loss": 3.5348,
+      "step": 77
+    },
+    {
+      "epoch": 0.014638265928497702,
+      "grad_norm": 4.596226692199707,
+      "learning_rate": 0.00019999552750959992,
+      "loss": 3.0881,
+      "step": 78
+    },
+    {
+      "epoch": 0.014825936004504082,
+      "grad_norm": 7.312027454376221,
+      "learning_rate": 0.00019999535288661718,
+      "loss": 3.08,
+      "step": 79
+    },
+    {
+      "epoch": 0.015013606080510463,
+      "grad_norm": 7.889541149139404,
+      "learning_rate": 0.000199995149159804,
+      "loss": 3.7032,
+      "step": 80
+    },
+    {
+      "epoch": 0.015201276156516843,
+      "grad_norm": 4.969526290893555,
+      "learning_rate": 0.0001999949454329908,
+      "loss": 3.4522,
+      "step": 81
+    },
+    {
+      "epoch": 0.015388946232523223,
+      "grad_norm": 4.675045967102051,
+      "learning_rate": 0.00019999475625809282,
+      "loss": 2.4161,
+      "step": 82
+    },
+    {
+      "epoch": 0.015576616308529604,
+      "grad_norm": 10.340566635131836,
+      "learning_rate": 0.00019999455253127962,
+      "loss": 3.3004,
+      "step": 83
+    },
+    {
+      "epoch": 0.015764286384535985,
+      "grad_norm": 6.192234516143799,
+      "learning_rate": 0.00019999434880446643,
+      "loss": 3.1927,
+      "step": 84
+    },
+    {
+      "epoch": 0.015951956460542368,
+      "grad_norm": 10.28073501586914,
+      "learning_rate": 0.000199994130525738,
+      "loss": 4.0849,
+      "step": 85
+    },
+    {
+      "epoch": 0.016139626536548747,
+      "grad_norm": 6.606441974639893,
+      "learning_rate": 0.00019999391224700958,
+      "loss": 2.946,
+      "step": 86
+    },
+    {
+      "epoch": 0.01632729661255513,
+      "grad_norm": 6.775887489318848,
+      "learning_rate": 0.00019999369396828115,
+      "loss": 3.0977,
+      "step": 87
+    },
+    {
+      "epoch": 0.01651496668856151,
+      "grad_norm": 7.575616836547852,
+      "learning_rate": 0.00019999347568955272,
+      "loss": 3.2857,
+      "step": 88
+    },
+    {
+      "epoch": 0.016702636764567888,
+      "grad_norm": 5.559998512268066,
+      "learning_rate": 0.00019999324285890907,
+      "loss": 2.5114,
+      "step": 89
+    },
+    {
+      "epoch": 0.01689030684057427,
+      "grad_norm": 26.611101150512695,
+      "learning_rate": 0.00019999302458018064,
+      "loss": 4.8027,
+      "step": 90
+    },
+    {
+      "epoch": 0.01707797691658065,
+      "grad_norm": 5.6725687980651855,
+      "learning_rate": 0.00019999277719762176,
+      "loss": 2.6003,
+      "step": 91
+    },
+    {
+      "epoch": 0.017265646992587032,
+      "grad_norm": 4.8655619621276855,
+      "learning_rate": 0.0001999925443669781,
+      "loss": 3.1824,
+      "step": 92
+    },
+    {
+      "epoch": 0.01745331706859341,
+      "grad_norm": 6.352896690368652,
+      "learning_rate": 0.00019999231153633446,
+      "loss": 1.9204,
+      "step": 93
+    },
+    {
+      "epoch": 0.017640987144599794,
+      "grad_norm": 10.785133361816406,
+      "learning_rate": 0.00019999204960186034,
+      "loss": 3.3155,
+      "step": 94
+    },
+    {
+      "epoch": 0.017828657220606173,
+      "grad_norm": 5.918345928192139,
+      "learning_rate": 0.00019999180221930146,
+      "loss": 3.1234,
+      "step": 95
+    },
+    {
+      "epoch": 0.018016327296612556,
+      "grad_norm": 7.9781341552734375,
+      "learning_rate": 0.00019999155483674258,
+      "loss": 4.4409,
+      "step": 96
+    },
+    {
+      "epoch": 0.018203997372618935,
+      "grad_norm": 6.567505836486816,
+      "learning_rate": 0.00019999129290226847,
+      "loss": 3.2331,
+      "step": 97
+    },
+    {
+      "epoch": 0.018391667448625318,
+      "grad_norm": 6.682971477508545,
+      "learning_rate": 0.00019999103096779436,
+      "loss": 3.1438,
+      "step": 98
+    },
+    {
+      "epoch": 0.018579337524631697,
+      "grad_norm": 5.597483158111572,
+      "learning_rate": 0.00019999076903332025,
+      "loss": 3.2274,
+      "step": 99
+    },
+    {
+      "epoch": 0.01876700760063808,
+      "grad_norm": 7.108291149139404,
+      "learning_rate": 0.0001999904925469309,
+      "loss": 3.3527,
+      "step": 100
+    },
+    {
+      "epoch": 0.01876700760063808,
+      "eval_loss": 0.864631175994873,
+      "eval_runtime": 60.8901,
+      "eval_samples_per_second": 3.531,
+      "eval_steps_per_second": 1.774,
+      "step": 100
+    }
+  ],
+  "logging_steps": 1,
+  "max_steps": 15984,
+  "num_input_tokens_seen": 0,
+  "num_train_epochs": 3,
+  "save_steps": 100,
+  "stateful_callbacks": {
+    "EarlyStoppingCallback": {
+      "args": {
+        "early_stopping_patience": 4,
+        "early_stopping_threshold": 0.0
+      },
+      "attributes": {
+        "early_stopping_patience_counter": 0
+      }
+    },
+    "TrainerControl": {
+      "args": {
+        "should_epoch_stop": false,
+        "should_evaluate": false,
+        "should_log": false,
+        "should_save": true,
+        "should_training_stop": false
+      },
+      "attributes": {}
+    }
+  },
+  "total_flos": 1.7882659619383296e+16,
+  "train_batch_size": 2,
+  "trial_name": null,
+  "trial_params": null
+}

last-checkpoint/training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:56c563ae444bf00c9513ce74b17ffcec432158c4f23af86b19e13547cfb69ea5
+size 6776