Training in progress, epoch 9, checkpoint

checkpoint-32391/README.md

@@ -0,0 +1,207 @@
+---
+base_model: deepseek-ai/deepseek-coder-6.7b-base
+library_name: peft
+pipeline_tag: text-generation
+tags:
+- base_model:adapter:deepseek-ai/deepseek-coder-6.7b-base
+- lora
+- transformers
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.17.1

checkpoint-32391/adapter_config.json

@@ -0,0 +1,37 @@
+{
+  "alpha_pattern": {},
+  "auto_mapping": null,
+  "base_model_name_or_path": "deepseek-ai/deepseek-coder-6.7b-base",
+  "bias": "none",
+  "corda_config": null,
+  "eva_config": null,
+  "exclude_modules": null,
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 32,
+  "lora_bias": false,
+  "lora_dropout": 0.05,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "qalora_group_size": 16,
+  "r": 8,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "q_proj",
+    "v_proj"
+  ],
+  "target_parameters": null,
+  "task_type": "CAUSAL_LM",
+  "trainable_token_indices": null,
+  "use_dora": false,
+  "use_qalora": false,
+  "use_rslora": false
+}

checkpoint-32391/adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1f00435676f7fffb05d69b9a7c3b1ab48c79ce34b56b72d09f82c75dd93d3353
+size 541442872

checkpoint-32391/optimizer.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f15ee488c31874f4202f78a062224b19651389ded9ffacda4ef34ee608546600
+size 33664331

checkpoint-32391/rng_state_0.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:03fa521d2cbd66723765bf0ab5231360cd04e59b025a7e8fb8e443e47d740e16
+size 15429

checkpoint-32391/rng_state_1.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1e950b7b4d92b5c1b9a5f636634a37e641ba413a7016ec518aeb3b885292a60b
+size 15429

checkpoint-32391/rng_state_2.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:323e686e1200c4b1f111c674ee7c09ab3b0de5ee8a536b083bbc62c87fa45a02
+size 15429

checkpoint-32391/rng_state_3.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4e9f8c66b5bb25c45f66fc4ef8cc3e5081c6f2b083112f3a2b45ff1d079aee51
+size 15429

checkpoint-32391/scaler.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:68e7755116e5a328e3b0591c5c7b4e2b055e71a1d3865d416b158778f49a17e2
+size 1383

checkpoint-32391/scheduler.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:71689cf3d1f55403f30c8c27c317a6e19ed65385b7b120ae60ba4bf513442a7b
+size 1465

checkpoint-32391/special_tokens_map.json

@@ -0,0 +1,23 @@
+{
+  "bos_token": {
+    "content": "<｜begin▁of▁sentence｜>",
+    "lstrip": false,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "<｜end▁of▁sentence｜>",
+    "lstrip": false,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "<｜end▁of▁sentence｜>",
+    "lstrip": false,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  }
+}

checkpoint-32391/tokenizer_config.json

@@ -0,0 +1,194 @@
+{
+  "add_bos_token": true,
+  "add_eos_token": false,
+  "add_prefix_space": null,
+  "added_tokens_decoder": {
+    "32000": {
+      "content": "õ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "32001": {
+      "content": "÷",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "32002": {
+      "content": "Á",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "32003": {
+      "content": "ý",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "32004": {
+      "content": "À",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "32005": {
+      "content": "ÿ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "32006": {
+      "content": "ø",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "32007": {
+      "content": "ú",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "32008": {
+      "content": "þ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "32009": {
+      "content": "ü",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "32010": {
+      "content": "ù",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "32011": {
+      "content": "ö",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "32012": {
+      "content": "û",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "32013": {
+      "content": "<｜begin▁of▁sentence｜>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "32014": {
+      "content": "<｜end▁of▁sentence｜>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "32015": {
+      "content": "<｜fim▁hole｜>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "32016": {
+      "content": "<｜fim▁begin｜>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "32017": {
+      "content": "<｜fim▁end｜>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "32018": {
+      "content": "<pad>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "32019": {
+      "content": "<|User|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "32020": {
+      "content": "<|Assistant|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "32021": {
+      "content": "<|EOT|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    }
+  },
+  "bos_token": "<｜begin▁of▁sentence｜>",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<｜end▁of▁sentence｜>",
+  "extra_special_tokens": {},
+  "legacy": true,
+  "model_max_length": 16384,
+  "pad_token": "<｜end▁of▁sentence｜>",
+  "sp_model_kwargs": {},
+  "tokenizer_class": "LlamaTokenizerFast",
+  "unk_token": null,
+  "use_default_system_prompt": false
+}

checkpoint-32391/trainer_state.json

@@ -0,0 +1,834 @@
+{
+  "best_global_step": null,
+  "best_metric": null,
+  "best_model_checkpoint": null,
+  "epoch": 9.0,
+  "eval_steps": 720,
+  "global_step": 32391,
+  "is_hyper_param_search": false,
+  "is_local_process_zero": true,
+  "is_world_process_zero": true,
+  "log_history": [
+    {
+      "epoch": 0.13892747985551543,
+      "grad_norm": 0.6049483418464661,
+      "learning_rate": 0.00039537834583680656,
+      "loss": 0.4045,
+      "step": 500
+    },
+    {
+      "epoch": 0.2000555709919422,
+      "eval_loss": 0.3253306448459625,
+      "eval_runtime": 12.9469,
+      "eval_samples_per_second": 38.619,
+      "eval_steps_per_second": 4.866,
+      "step": 720
+    },
+    {
+      "epoch": 0.27785495971103086,
+      "grad_norm": 0.6276679039001465,
+      "learning_rate": 0.0003907474298416227,
+      "loss": 0.3156,
+      "step": 1000
+    },
+    {
+      "epoch": 0.4001111419838844,
+      "eval_loss": 0.29051902890205383,
+      "eval_runtime": 12.6392,
+      "eval_samples_per_second": 39.56,
+      "eval_steps_per_second": 4.985,
+      "step": 1440
+    },
+    {
+      "epoch": 0.41678243956654626,
+      "grad_norm": 0.6468805074691772,
+      "learning_rate": 0.00038611651384643886,
+      "loss": 0.2857,
+      "step": 1500
+    },
+    {
+      "epoch": 0.5557099194220617,
+      "grad_norm": 0.6646202206611633,
+      "learning_rate": 0.000381485597851255,
+      "loss": 0.2636,
+      "step": 2000
+    },
+    {
+      "epoch": 0.6001667129758266,
+      "eval_loss": 0.2643604874610901,
+      "eval_runtime": 12.8691,
+      "eval_samples_per_second": 38.853,
+      "eval_steps_per_second": 4.895,
+      "step": 2160
+    },
+    {
+      "epoch": 0.6946373992775771,
+      "grad_norm": 0.512627124786377,
+      "learning_rate": 0.00037685468185607115,
+      "loss": 0.2455,
+      "step": 2500
+    },
+    {
+      "epoch": 0.8002222839677688,
+      "eval_loss": 0.24014389514923096,
+      "eval_runtime": 12.6518,
+      "eval_samples_per_second": 39.52,
+      "eval_steps_per_second": 4.98,
+      "step": 2880
+    },
+    {
+      "epoch": 0.8335648791330925,
+      "grad_norm": 0.5946218371391296,
+      "learning_rate": 0.0003722237658608873,
+      "loss": 0.232,
+      "step": 3000
+    },
+    {
+      "epoch": 0.972492358988608,
+      "grad_norm": 0.6397203803062439,
+      "learning_rate": 0.00036759284986570345,
+      "loss": 0.227,
+      "step": 3500
+    },
+    {
+      "epoch": 1.000277854959711,
+      "eval_loss": 0.23284229636192322,
+      "eval_runtime": 13.3745,
+      "eval_samples_per_second": 37.385,
+      "eval_steps_per_second": 4.71,
+      "step": 3600
+    },
+    {
+      "epoch": 1.1114198388441234,
+      "grad_norm": 0.9304324984550476,
+      "learning_rate": 0.0003629619338705196,
+      "loss": 0.1903,
+      "step": 4000
+    },
+    {
+      "epoch": 1.2003334259516532,
+      "eval_loss": 0.22673270106315613,
+      "eval_runtime": 12.5927,
+      "eval_samples_per_second": 39.706,
+      "eval_steps_per_second": 5.003,
+      "step": 4320
+    },
+    {
+      "epoch": 1.2503473186996388,
+      "grad_norm": 0.7105266451835632,
+      "learning_rate": 0.00035833101787533574,
+      "loss": 0.1897,
+      "step": 4500
+    },
+    {
+      "epoch": 1.3892747985551543,
+      "grad_norm": 0.8217107057571411,
+      "learning_rate": 0.0003537001018801519,
+      "loss": 0.1872,
+      "step": 5000
+    },
+    {
+      "epoch": 1.4003889969435954,
+      "eval_loss": 0.21865838766098022,
+      "eval_runtime": 12.5947,
+      "eval_samples_per_second": 39.699,
+      "eval_steps_per_second": 5.002,
+      "step": 5040
+    },
+    {
+      "epoch": 1.5282022784106695,
+      "grad_norm": 0.664078414440155,
+      "learning_rate": 0.00034906918588496804,
+      "loss": 0.1831,
+      "step": 5500
+    },
+    {
+      "epoch": 1.6004445679355377,
+      "eval_loss": 0.2104523777961731,
+      "eval_runtime": 12.7945,
+      "eval_samples_per_second": 39.079,
+      "eval_steps_per_second": 4.924,
+      "step": 5760
+    },
+    {
+      "epoch": 1.667129758266185,
+      "grad_norm": 0.6712161898612976,
+      "learning_rate": 0.0003444382698897842,
+      "loss": 0.1785,
+      "step": 6000
+    },
+    {
+      "epoch": 1.8005001389274797,
+      "eval_loss": 0.20817357301712036,
+      "eval_runtime": 12.7871,
+      "eval_samples_per_second": 39.102,
+      "eval_steps_per_second": 4.927,
+      "step": 6480
+    },
+    {
+      "epoch": 1.8060572381217006,
+      "grad_norm": 0.5696629285812378,
+      "learning_rate": 0.0003398073538946004,
+      "loss": 0.1737,
+      "step": 6500
+    },
+    {
+      "epoch": 1.9449847179772157,
+      "grad_norm": 0.5998601913452148,
+      "learning_rate": 0.0003351764378994165,
+      "loss": 0.1736,
+      "step": 7000
+    },
+    {
+      "epoch": 2.000555709919422,
+      "eval_loss": 0.19734103977680206,
+      "eval_runtime": 12.6069,
+      "eval_samples_per_second": 39.661,
+      "eval_steps_per_second": 4.997,
+      "step": 7200
+    },
+    {
+      "epoch": 2.0839121978327313,
+      "grad_norm": 0.8763924241065979,
+      "learning_rate": 0.0003305455219042327,
+      "loss": 0.1512,
+      "step": 7500
+    },
+    {
+      "epoch": 2.2006112809113643,
+      "eval_loss": 0.1998797059059143,
+      "eval_runtime": 12.5576,
+      "eval_samples_per_second": 39.816,
+      "eval_steps_per_second": 5.017,
+      "step": 7920
+    },
+    {
+      "epoch": 2.222839677688247,
+      "grad_norm": 0.8139877915382385,
+      "learning_rate": 0.0003259146059090488,
+      "loss": 0.1403,
+      "step": 8000
+    },
+    {
+      "epoch": 2.361767157543762,
+      "grad_norm": 0.6765368580818176,
+      "learning_rate": 0.000321283689913865,
+      "loss": 0.1414,
+      "step": 8500
+    },
+    {
+      "epoch": 2.4006668519033063,
+      "eval_loss": 0.19732815027236938,
+      "eval_runtime": 12.6446,
+      "eval_samples_per_second": 39.543,
+      "eval_steps_per_second": 4.982,
+      "step": 8640
+    },
+    {
+      "epoch": 2.5006946373992776,
+      "grad_norm": 0.6791777014732361,
+      "learning_rate": 0.00031665277391868115,
+      "loss": 0.1413,
+      "step": 9000
+    },
+    {
+      "epoch": 2.600722422895249,
+      "eval_loss": 0.18912270665168762,
+      "eval_runtime": 12.7219,
+      "eval_samples_per_second": 39.302,
+      "eval_steps_per_second": 4.952,
+      "step": 9360
+    },
+    {
+      "epoch": 2.639622117254793,
+      "grad_norm": 0.7267652750015259,
+      "learning_rate": 0.00031202185792349727,
+      "loss": 0.139,
+      "step": 9500
+    },
+    {
+      "epoch": 2.7785495971103087,
+      "grad_norm": 0.4005363881587982,
+      "learning_rate": 0.0003073909419283134,
+      "loss": 0.1397,
+      "step": 10000
+    },
+    {
+      "epoch": 2.800777993887191,
+      "eval_loss": 0.1857079565525055,
+      "eval_runtime": 12.6567,
+      "eval_samples_per_second": 39.505,
+      "eval_steps_per_second": 4.978,
+      "step": 10080
+    },
+    {
+      "epoch": 2.917477076965824,
+      "grad_norm": 0.8150336742401123,
+      "learning_rate": 0.00030276002593312957,
+      "loss": 0.1401,
+      "step": 10500
+    },
+    {
+      "epoch": 3.000833564879133,
+      "eval_loss": 0.18395407497882843,
+      "eval_runtime": 12.6261,
+      "eval_samples_per_second": 39.6,
+      "eval_steps_per_second": 4.99,
+      "step": 10800
+    },
+    {
+      "epoch": 3.0564045568213394,
+      "grad_norm": 0.6333420276641846,
+      "learning_rate": 0.00029812910993794574,
+      "loss": 0.1262,
+      "step": 11000
+    },
+    {
+      "epoch": 3.1953320366768545,
+      "grad_norm": 0.46257224678993225,
+      "learning_rate": 0.0002934981939427619,
+      "loss": 0.1111,
+      "step": 11500
+    },
+    {
+      "epoch": 3.2008891358710754,
+      "eval_loss": 0.18369066715240479,
+      "eval_runtime": 12.6247,
+      "eval_samples_per_second": 39.605,
+      "eval_steps_per_second": 4.99,
+      "step": 11520
+    },
+    {
+      "epoch": 3.33425951653237,
+      "grad_norm": 0.7679395079612732,
+      "learning_rate": 0.00028886727794757804,
+      "loss": 0.1135,
+      "step": 12000
+    },
+    {
+      "epoch": 3.4009447068630174,
+      "eval_loss": 0.18193913996219635,
+      "eval_runtime": 12.625,
+      "eval_samples_per_second": 39.604,
+      "eval_steps_per_second": 4.99,
+      "step": 12240
+    },
+    {
+      "epoch": 3.4731869963878856,
+      "grad_norm": 0.576283872127533,
+      "learning_rate": 0.00028423636195239416,
+      "loss": 0.1138,
+      "step": 12500
+    },
+    {
+      "epoch": 3.6010002778549595,
+      "eval_loss": 0.18214693665504456,
+      "eval_runtime": 12.6409,
+      "eval_samples_per_second": 39.554,
+      "eval_steps_per_second": 4.984,
+      "step": 12960
+    },
+    {
+      "epoch": 3.612114476243401,
+      "grad_norm": 0.8275533318519592,
+      "learning_rate": 0.00027960544595721033,
+      "loss": 0.1148,
+      "step": 13000
+    },
+    {
+      "epoch": 3.7510419560989163,
+      "grad_norm": 0.7274850010871887,
+      "learning_rate": 0.0002749745299620265,
+      "loss": 0.1133,
+      "step": 13500
+    },
+    {
+      "epoch": 3.801055848846902,
+      "eval_loss": 0.17986078560352325,
+      "eval_runtime": 12.6956,
+      "eval_samples_per_second": 39.384,
+      "eval_steps_per_second": 4.962,
+      "step": 13680
+    },
+    {
+      "epoch": 3.889969435954432,
+      "grad_norm": 0.6496698260307312,
+      "learning_rate": 0.0002703436139668427,
+      "loss": 0.1147,
+      "step": 14000
+    },
+    {
+      "epoch": 4.001111419838844,
+      "eval_loss": 0.1769217699766159,
+      "eval_runtime": 12.661,
+      "eval_samples_per_second": 39.491,
+      "eval_steps_per_second": 4.976,
+      "step": 14400
+    },
+    {
+      "epoch": 4.0288969158099475,
+      "grad_norm": 0.72037273645401,
+      "learning_rate": 0.0002657126979716588,
+      "loss": 0.1069,
+      "step": 14500
+    },
+    {
+      "epoch": 4.167824395665463,
+      "grad_norm": 0.7723105549812317,
+      "learning_rate": 0.0002610817819764749,
+      "loss": 0.0871,
+      "step": 15000
+    },
+    {
+      "epoch": 4.201166990830786,
+      "eval_loss": 0.1821662336587906,
+      "eval_runtime": 12.6066,
+      "eval_samples_per_second": 39.662,
+      "eval_steps_per_second": 4.997,
+      "step": 15120
+    },
+    {
+      "epoch": 4.306751875520978,
+      "grad_norm": 0.5981574654579163,
+      "learning_rate": 0.0002564508659812911,
+      "loss": 0.0902,
+      "step": 15500
+    },
+    {
+      "epoch": 4.4012225618227285,
+      "eval_loss": 0.18598410487174988,
+      "eval_runtime": 12.5969,
+      "eval_samples_per_second": 39.692,
+      "eval_steps_per_second": 5.001,
+      "step": 15840
+    },
+    {
+      "epoch": 4.445679355376494,
+      "grad_norm": 0.7958341240882874,
+      "learning_rate": 0.00025181994998610727,
+      "loss": 0.0915,
+      "step": 16000
+    },
+    {
+      "epoch": 4.584606835232009,
+      "grad_norm": 0.6806578636169434,
+      "learning_rate": 0.00024718903399092345,
+      "loss": 0.0921,
+      "step": 16500
+    },
+    {
+      "epoch": 4.601278132814671,
+      "eval_loss": 0.1808934509754181,
+      "eval_runtime": 12.8476,
+      "eval_samples_per_second": 38.918,
+      "eval_steps_per_second": 4.904,
+      "step": 16560
+    },
+    {
+      "epoch": 4.723534315087524,
+      "grad_norm": 0.9893471002578735,
+      "learning_rate": 0.00024255811799573954,
+      "loss": 0.0956,
+      "step": 17000
+    },
+    {
+      "epoch": 4.801333703806613,
+      "eval_loss": 0.16961801052093506,
+      "eval_runtime": 12.6325,
+      "eval_samples_per_second": 39.58,
+      "eval_steps_per_second": 4.987,
+      "step": 17280
+    },
+    {
+      "epoch": 4.86246179494304,
+      "grad_norm": 0.4688684344291687,
+      "learning_rate": 0.00023792720200055571,
+      "loss": 0.0951,
+      "step": 17500
+    },
+    {
+      "epoch": 5.001389274798555,
+      "grad_norm": 0.644500195980072,
+      "learning_rate": 0.0002332962860053719,
+      "loss": 0.0932,
+      "step": 18000
+    },
+    {
+      "epoch": 5.001389274798555,
+      "eval_loss": 0.17107965052127838,
+      "eval_runtime": 12.6248,
+      "eval_samples_per_second": 39.605,
+      "eval_steps_per_second": 4.99,
+      "step": 18000
+    },
+    {
+      "epoch": 5.14031675465407,
+      "grad_norm": 0.4825419783592224,
+      "learning_rate": 0.00022866537001018804,
+      "loss": 0.0706,
+      "step": 18500
+    },
+    {
+      "epoch": 5.201444845790498,
+      "eval_loss": 0.17916250228881836,
+      "eval_runtime": 12.632,
+      "eval_samples_per_second": 39.582,
+      "eval_steps_per_second": 4.987,
+      "step": 18720
+    },
+    {
+      "epoch": 5.279244234509586,
+      "grad_norm": 0.49617692828178406,
+      "learning_rate": 0.00022403445401500416,
+      "loss": 0.0715,
+      "step": 19000
+    },
+    {
+      "epoch": 5.401500416782439,
+      "eval_loss": 0.18028731644153595,
+      "eval_runtime": 12.6155,
+      "eval_samples_per_second": 39.634,
+      "eval_steps_per_second": 4.994,
+      "step": 19440
+    },
+    {
+      "epoch": 5.418171714365101,
+      "grad_norm": 0.5684732794761658,
+      "learning_rate": 0.00021940353801982033,
+      "loss": 0.0737,
+      "step": 19500
+    },
+    {
+      "epoch": 5.5570991942206165,
+      "grad_norm": 0.7372530698776245,
+      "learning_rate": 0.00021477262202463648,
+      "loss": 0.075,
+      "step": 20000
+    },
+    {
+      "epoch": 5.601555987774382,
+      "eval_loss": 0.17456859350204468,
+      "eval_runtime": 12.9144,
+      "eval_samples_per_second": 38.717,
+      "eval_steps_per_second": 4.878,
+      "step": 20160
+    },
+    {
+      "epoch": 5.6960266740761325,
+      "grad_norm": 0.663422703742981,
+      "learning_rate": 0.00021014170602945265,
+      "loss": 0.0775,
+      "step": 20500
+    },
+    {
+      "epoch": 5.801611558766324,
+      "eval_loss": 0.17978759109973907,
+      "eval_runtime": 12.6657,
+      "eval_samples_per_second": 39.477,
+      "eval_steps_per_second": 4.974,
+      "step": 20880
+    },
+    {
+      "epoch": 5.834954153931648,
+      "grad_norm": 1.133213758468628,
+      "learning_rate": 0.0002055107900342688,
+      "loss": 0.0757,
+      "step": 21000
+    },
+    {
+      "epoch": 5.973881633787163,
+      "grad_norm": 0.6035337448120117,
+      "learning_rate": 0.00020087987403908492,
+      "loss": 0.0759,
+      "step": 21500
+    },
+    {
+      "epoch": 6.001667129758266,
+      "eval_loss": 0.176605224609375,
+      "eval_runtime": 12.6501,
+      "eval_samples_per_second": 39.525,
+      "eval_steps_per_second": 4.98,
+      "step": 21600
+    },
+    {
+      "epoch": 6.112809113642679,
+      "grad_norm": 0.5326036214828491,
+      "learning_rate": 0.0001962489580439011,
+      "loss": 0.0604,
+      "step": 22000
+    },
+    {
+      "epoch": 6.201722700750208,
+      "eval_loss": 0.18489301204681396,
+      "eval_runtime": 12.7563,
+      "eval_samples_per_second": 39.196,
+      "eval_steps_per_second": 4.939,
+      "step": 22320
+    },
+    {
+      "epoch": 6.251736593498194,
+      "grad_norm": 0.7067334651947021,
+      "learning_rate": 0.00019161804204871724,
+      "loss": 0.0577,
+      "step": 22500
+    },
+    {
+      "epoch": 6.390664073353709,
+      "grad_norm": 0.5834971070289612,
+      "learning_rate": 0.0001869871260535334,
+      "loss": 0.0608,
+      "step": 23000
+    },
+    {
+      "epoch": 6.401778271742151,
+      "eval_loss": 0.1874849945306778,
+      "eval_runtime": 12.9554,
+      "eval_samples_per_second": 38.594,
+      "eval_steps_per_second": 4.863,
+      "step": 23040
+    },
+    {
+      "epoch": 6.529591553209225,
+      "grad_norm": 0.776667058467865,
+      "learning_rate": 0.00018235621005834954,
+      "loss": 0.0626,
+      "step": 23500
+    },
+    {
+      "epoch": 6.601833842734093,
+      "eval_loss": 0.17743869125843048,
+      "eval_runtime": 12.7734,
+      "eval_samples_per_second": 39.144,
+      "eval_steps_per_second": 4.932,
+      "step": 23760
+    },
+    {
+      "epoch": 6.66851903306474,
+      "grad_norm": 0.607370913028717,
+      "learning_rate": 0.0001777252940631657,
+      "loss": 0.0614,
+      "step": 24000
+    },
+    {
+      "epoch": 6.801889413726035,
+      "eval_loss": 0.17762653529644012,
+      "eval_runtime": 12.6343,
+      "eval_samples_per_second": 39.575,
+      "eval_steps_per_second": 4.986,
+      "step": 24480
+    },
+    {
+      "epoch": 6.807446512920255,
+      "grad_norm": 0.442354291677475,
+      "learning_rate": 0.00017309437806798186,
+      "loss": 0.0623,
+      "step": 24500
+    },
+    {
+      "epoch": 6.946373992775771,
+      "grad_norm": 0.5223565101623535,
+      "learning_rate": 0.000168463462072798,
+      "loss": 0.0622,
+      "step": 25000
+    },
+    {
+      "epoch": 7.001944984717977,
+      "eval_loss": 0.1797703206539154,
+      "eval_runtime": 12.6168,
+      "eval_samples_per_second": 39.63,
+      "eval_steps_per_second": 4.993,
+      "step": 25200
+    },
+    {
+      "epoch": 7.085301472631286,
+      "grad_norm": 0.4835590124130249,
+      "learning_rate": 0.00016383254607761416,
+      "loss": 0.0505,
+      "step": 25500
+    },
+    {
+      "epoch": 7.20200055570992,
+      "eval_loss": 0.19183334708213806,
+      "eval_runtime": 12.6054,
+      "eval_samples_per_second": 39.666,
+      "eval_steps_per_second": 4.998,
+      "step": 25920
+    },
+    {
+      "epoch": 7.2242289524868015,
+      "grad_norm": 0.5445060133934021,
+      "learning_rate": 0.00015920163008243033,
+      "loss": 0.0452,
+      "step": 26000
+    },
+    {
+      "epoch": 7.3631564323423175,
+      "grad_norm": 0.5222957134246826,
+      "learning_rate": 0.00015457071408724645,
+      "loss": 0.0475,
+      "step": 26500
+    },
+    {
+      "epoch": 7.402056126701861,
+      "eval_loss": 0.19407090544700623,
+      "eval_runtime": 12.5977,
+      "eval_samples_per_second": 39.69,
+      "eval_steps_per_second": 5.001,
+      "step": 26640
+    },
+    {
+      "epoch": 7.502083912197833,
+      "grad_norm": 0.46540552377700806,
+      "learning_rate": 0.00014993979809206263,
+      "loss": 0.0487,
+      "step": 27000
+    },
+    {
+      "epoch": 7.602111697693804,
+      "eval_loss": 0.18855853378772736,
+      "eval_runtime": 12.5796,
+      "eval_samples_per_second": 39.747,
+      "eval_steps_per_second": 5.008,
+      "step": 27360
+    },
+    {
+      "epoch": 7.641011392053348,
+      "grad_norm": 0.5198289155960083,
+      "learning_rate": 0.00014530888209687877,
+      "loss": 0.049,
+      "step": 27500
+    },
+    {
+      "epoch": 7.779938871908864,
+      "grad_norm": 0.6500052213668823,
+      "learning_rate": 0.00014067796610169492,
+      "loss": 0.0507,
+      "step": 28000
+    },
+    {
+      "epoch": 7.802167268685746,
+      "eval_loss": 0.18595577776432037,
+      "eval_runtime": 12.5971,
+      "eval_samples_per_second": 39.692,
+      "eval_steps_per_second": 5.001,
+      "step": 28080
+    },
+    {
+      "epoch": 7.918866351764379,
+      "grad_norm": 0.7434415221214294,
+      "learning_rate": 0.00013604705010651107,
+      "loss": 0.0507,
+      "step": 28500
+    },
+    {
+      "epoch": 8.002222839677689,
+      "eval_loss": 0.1883331686258316,
+      "eval_runtime": 12.6223,
+      "eval_samples_per_second": 39.612,
+      "eval_steps_per_second": 4.991,
+      "step": 28800
+    },
+    {
+      "epoch": 8.057793831619895,
+      "grad_norm": 0.8438473343849182,
+      "learning_rate": 0.00013141613411132722,
+      "loss": 0.0427,
+      "step": 29000
+    },
+    {
+      "epoch": 8.19672131147541,
+      "grad_norm": 0.44807878136634827,
+      "learning_rate": 0.0001267852181161434,
+      "loss": 0.0372,
+      "step": 29500
+    },
+    {
+      "epoch": 8.20227841066963,
+      "eval_loss": 0.2032124102115631,
+      "eval_runtime": 12.7528,
+      "eval_samples_per_second": 39.207,
+      "eval_steps_per_second": 4.94,
+      "step": 29520
+    },
+    {
+      "epoch": 8.335648791330925,
+      "grad_norm": 0.6269062757492065,
+      "learning_rate": 0.00012215430212095954,
+      "loss": 0.0368,
+      "step": 30000
+    },
+    {
+      "epoch": 8.402333981661572,
+      "eval_loss": 0.19736090302467346,
+      "eval_runtime": 12.6619,
+      "eval_samples_per_second": 39.489,
+      "eval_steps_per_second": 4.976,
+      "step": 30240
+    },
+    {
+      "epoch": 8.474576271186441,
+      "grad_norm": 0.5166807174682617,
+      "learning_rate": 0.00011752338612577569,
+      "loss": 0.0383,
+      "step": 30500
+    },
+    {
+      "epoch": 8.602389552653515,
+      "eval_loss": 0.1965281069278717,
+      "eval_runtime": 12.6481,
+      "eval_samples_per_second": 39.532,
+      "eval_steps_per_second": 4.981,
+      "step": 30960
+    },
+    {
+      "epoch": 8.613503751041955,
+      "grad_norm": 0.5574499368667603,
+      "learning_rate": 0.00011289247013059183,
+      "loss": 0.0389,
+      "step": 31000
+    },
+    {
+      "epoch": 8.752431230897471,
+      "grad_norm": 0.6240259408950806,
+      "learning_rate": 0.000108261554135408,
+      "loss": 0.0387,
+      "step": 31500
+    },
+    {
+      "epoch": 8.802445123645457,
+      "eval_loss": 0.19709058105945587,
+      "eval_runtime": 12.6201,
+      "eval_samples_per_second": 39.619,
+      "eval_steps_per_second": 4.992,
+      "step": 31680
+    },
+    {
+      "epoch": 8.891358710752987,
+      "grad_norm": 0.8563889265060425,
+      "learning_rate": 0.00010363063814022413,
+      "loss": 0.0395,
+      "step": 32000
+    }
+  ],
+  "logging_steps": 500,
+  "max_steps": 43188,
+  "num_input_tokens_seen": 0,
+  "num_train_epochs": 12,
+  "save_steps": 500,
+  "stateful_callbacks": {
+    "TrainerControl": {
+      "args": {
+        "should_epoch_stop": false,
+        "should_evaluate": false,
+        "should_log": false,
+        "should_save": true,
+        "should_training_stop": false
+      },
+      "attributes": {}
+    }
+  },
+  "total_flos": 5.663978223310995e+18,
+  "train_batch_size": 4,
+  "trial_name": null,
+  "trial_params": null
+}

checkpoint-32391/training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ea190ee64a0bc0efb2ac2058843e39eb424a57fffc6d8909debf7ec6d0496d71
+size 5969