Ajout dans checkpoints/

checkpoints/README.md

@@ -0,0 +1,202 @@
+---
+base_model: bigcode/starcoder2-3b
+library_name: peft
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.15.2.dev0

checkpoints/adapter_config.json

@@ -0,0 +1,34 @@
+{
+  "alpha_pattern": {},
+  "auto_mapping": null,
+  "base_model_name_or_path": "bigcode/starcoder2-3b",
+  "bias": "none",
+  "corda_config": null,
+  "eva_config": null,
+  "exclude_modules": null,
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 16,
+  "lora_bias": false,
+  "lora_dropout": 0.05,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "r": 8,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "v_proj",
+    "q_proj"
+  ],
+  "task_type": "CAUSAL_LM",
+  "trainable_token_indices": null,
+  "use_dora": false,
+  "use_rslora": false
+}

checkpoints/adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8afabc27017a9b3ed48cc1ac35a4073ef313ea55d598a40e0150936610e189bf
+size 9108904

checkpoints/optimizer.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7ccfd53dc0de98477a6681b297b76b88b4d13a786179e4ba8e3535c46c1044ef
+size 18287162

checkpoints/rng_state.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:63eb33e145e1c0ab82b446db9a32fc3ca8920bc98e3b1648d8ea6bcbf6d3e84e
+size 14244

checkpoints/scaler.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5f90fd3ad19bd4754603bfd3c6a498b8dcd94e4832597c952106bd7c9c3d40ec
+size 988

checkpoints/scheduler.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:23697061698ab6b2cd4d7a33dd4b6049a6abf090d53017154d8146e7b5c7353c
+size 1064

checkpoints/special_tokens_map.json

@@ -0,0 +1,63 @@
+{
+  "additional_special_tokens": [
+    "<|endoftext|>",
+    "<fim_prefix>",
+    "<fim_middle>",
+    "<fim_suffix>",
+    "<fim_pad>",
+    "<repo_name>",
+    "<file_sep>",
+    "<issue_start>",
+    "<issue_comment>",
+    "<issue_closed>",
+    "<jupyter_start>",
+    "<jupyter_text>",
+    "<jupyter_code>",
+    "<jupyter_output>",
+    "<jupyter_script>",
+    "<empty_output>",
+    "<code_to_intermediate>",
+    "<intermediate_to_code>",
+    "<pr>",
+    "<pr_status>",
+    "<pr_is_merged>",
+    "<pr_base>",
+    "<pr_file>",
+    "<pr_base_code>",
+    "<pr_diff>",
+    "<pr_diff_hunk>",
+    "<pr_comment>",
+    "<pr_event_id>",
+    "<pr_review>",
+    "<pr_review_state>",
+    "<pr_review_comment>",
+    "<pr_in_reply_to_review_id>",
+    "<pr_in_reply_to_comment_id>",
+    "<pr_diff_hunk_comment_line>",
+    "<NAME>",
+    "<EMAIL>",
+    "<KEY>",
+    "<PASSWORD>"
+  ],
+  "bos_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "unk_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

checkpoints/tokenizer_config.json

@@ -0,0 +1,357 @@
+{
+  "add_prefix_space": false,
+  "added_tokens_decoder": {
+    "0": {
+      "content": "<|endoftext|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "1": {
+      "content": "<fim_prefix>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "2": {
+      "content": "<fim_middle>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "3": {
+      "content": "<fim_suffix>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "4": {
+      "content": "<fim_pad>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "5": {
+      "content": "<repo_name>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "6": {
+      "content": "<file_sep>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "7": {
+      "content": "<issue_start>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "8": {
+      "content": "<issue_comment>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "9": {
+      "content": "<issue_closed>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "10": {
+      "content": "<jupyter_start>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "11": {
+      "content": "<jupyter_text>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "12": {
+      "content": "<jupyter_code>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "13": {
+      "content": "<jupyter_output>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "14": {
+      "content": "<jupyter_script>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "15": {
+      "content": "<empty_output>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "16": {
+      "content": "<code_to_intermediate>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "17": {
+      "content": "<intermediate_to_code>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "18": {
+      "content": "<pr>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "19": {
+      "content": "<pr_status>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "20": {
+      "content": "<pr_is_merged>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "21": {
+      "content": "<pr_base>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "22": {
+      "content": "<pr_file>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "23": {
+      "content": "<pr_base_code>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "24": {
+      "content": "<pr_diff>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "25": {
+      "content": "<pr_diff_hunk>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "26": {
+      "content": "<pr_comment>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "27": {
+      "content": "<pr_event_id>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "28": {
+      "content": "<pr_review>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "29": {
+      "content": "<pr_review_state>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "30": {
+      "content": "<pr_review_comment>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "31": {
+      "content": "<pr_in_reply_to_review_id>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "32": {
+      "content": "<pr_in_reply_to_comment_id>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "33": {
+      "content": "<pr_diff_hunk_comment_line>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "34": {
+      "content": "<NAME>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "35": {
+      "content": "<EMAIL>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "36": {
+      "content": "<KEY>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "37": {
+      "content": "<PASSWORD>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "additional_special_tokens": [
+    "<|endoftext|>",
+    "<fim_prefix>",
+    "<fim_middle>",
+    "<fim_suffix>",
+    "<fim_pad>",
+    "<repo_name>",
+    "<file_sep>",
+    "<issue_start>",
+    "<issue_comment>",
+    "<issue_closed>",
+    "<jupyter_start>",
+    "<jupyter_text>",
+    "<jupyter_code>",
+    "<jupyter_output>",
+    "<jupyter_script>",
+    "<empty_output>",
+    "<code_to_intermediate>",
+    "<intermediate_to_code>",
+    "<pr>",
+    "<pr_status>",
+    "<pr_is_merged>",
+    "<pr_base>",
+    "<pr_file>",
+    "<pr_base_code>",
+    "<pr_diff>",
+    "<pr_diff_hunk>",
+    "<pr_comment>",
+    "<pr_event_id>",
+    "<pr_review>",
+    "<pr_review_state>",
+    "<pr_review_comment>",
+    "<pr_in_reply_to_review_id>",
+    "<pr_in_reply_to_comment_id>",
+    "<pr_diff_hunk_comment_line>",
+    "<NAME>",
+    "<EMAIL>",
+    "<KEY>",
+    "<PASSWORD>"
+  ],
+  "bos_token": "<|endoftext|>",
+  "clean_up_tokenization_spaces": true,
+  "eos_token": "<|endoftext|>",
+  "extra_special_tokens": {},
+  "model_max_length": 1000000000000000019884624838656,
+  "tokenizer_class": "GPT2Tokenizer",
+  "unk_token": "<|endoftext|>",
+  "vocab_size": 49152
+}

checkpoints/trainer_state.json

@@ -0,0 +1,1501 @@
+{
+  "best_global_step": null,
+  "best_metric": null,
+  "best_model_checkpoint": null,
+  "epoch": 2.9949332105020727,
+  "eval_steps": 500,
+  "global_step": 1626,
+  "is_hyper_param_search": false,
+  "is_local_process_zero": true,
+  "is_world_process_zero": true,
+  "log_history": [
+    {
+      "epoch": 0.0018424689083371719,
+      "grad_norm": 1.5234806537628174,
+      "learning_rate": 0.0001,
+      "loss": 2.859,
+      "mean_token_accuracy": 0.4461010470986366,
+      "num_tokens": 1773.0,
+      "step": 1
+    },
+    {
+      "epoch": 0.018424689083371717,
+      "grad_norm": 1.7587577104568481,
+      "learning_rate": 9.944649446494465e-05,
+      "loss": 3.2432,
+      "mean_token_accuracy": 0.4386831981440385,
+      "num_tokens": 14075.0,
+      "step": 10
+    },
+    {
+      "epoch": 0.036849378166743434,
+      "grad_norm": 1.6922638416290283,
+      "learning_rate": 9.895448954489545e-05,
+      "loss": 3.2681,
+      "mean_token_accuracy": 0.4345304625108838,
+      "num_tokens": 30734.0,
+      "step": 20
+    },
+    {
+      "epoch": 0.055274067250115154,
+      "grad_norm": 1.5778034925460815,
+      "learning_rate": 9.833948339483395e-05,
+      "loss": 2.9895,
+      "mean_token_accuracy": 0.4445969723165035,
+      "num_tokens": 43216.0,
+      "step": 30
+    },
+    {
+      "epoch": 0.07369875633348687,
+      "grad_norm": 2.9133620262145996,
+      "learning_rate": 9.77859778597786e-05,
+      "loss": 3.0992,
+      "mean_token_accuracy": 0.4557989288121462,
+      "num_tokens": 59278.0,
+      "step": 40
+    },
+    {
+      "epoch": 0.09212344541685859,
+      "grad_norm": 1.233146071434021,
+      "learning_rate": 9.71709717097171e-05,
+      "loss": 2.6631,
+      "mean_token_accuracy": 0.4950188485905528,
+      "num_tokens": 76093.0,
+      "step": 50
+    },
+    {
+      "epoch": 0.11054813450023031,
+      "grad_norm": 1.8793283700942993,
+      "learning_rate": 9.65559655596556e-05,
+      "loss": 2.6269,
+      "mean_token_accuracy": 0.5195341780781746,
+      "num_tokens": 91394.0,
+      "step": 60
+    },
+    {
+      "epoch": 0.12897282358360201,
+      "grad_norm": 3.393233060836792,
+      "learning_rate": 9.59409594095941e-05,
+      "loss": 2.6286,
+      "mean_token_accuracy": 0.5205566212534904,
+      "num_tokens": 109148.0,
+      "step": 70
+    },
+    {
+      "epoch": 0.14739751266697373,
+      "grad_norm": 1.4595232009887695,
+      "learning_rate": 9.53259532595326e-05,
+      "loss": 2.4222,
+      "mean_token_accuracy": 0.5303220309317112,
+      "num_tokens": 124117.0,
+      "step": 80
+    },
+    {
+      "epoch": 0.16582220175034545,
+      "grad_norm": 1.319626808166504,
+      "learning_rate": 9.471094710947111e-05,
+      "loss": 2.6389,
+      "mean_token_accuracy": 0.5104626394808293,
+      "num_tokens": 138074.0,
+      "step": 90
+    },
+    {
+      "epoch": 0.18424689083371718,
+      "grad_norm": 1.364896535873413,
+      "learning_rate": 9.40959409594096e-05,
+      "loss": 2.5226,
+      "mean_token_accuracy": 0.5130734294652939,
+      "num_tokens": 151855.0,
+      "step": 100
+    },
+    {
+      "epoch": 0.2026715799170889,
+      "grad_norm": 3.1249070167541504,
+      "learning_rate": 9.34809348093481e-05,
+      "loss": 2.2236,
+      "mean_token_accuracy": 0.5498266529291869,
+      "num_tokens": 169671.0,
+      "step": 110
+    },
+    {
+      "epoch": 0.22109626900046062,
+      "grad_norm": 2.209862232208252,
+      "learning_rate": 9.28659286592866e-05,
+      "loss": 1.9976,
+      "mean_token_accuracy": 0.5757020100951195,
+      "num_tokens": 186277.0,
+      "step": 120
+    },
+    {
+      "epoch": 0.23952095808383234,
+      "grad_norm": 2.1681127548217773,
+      "learning_rate": 9.22509225092251e-05,
+      "loss": 2.2206,
+      "mean_token_accuracy": 0.5722524970769882,
+      "num_tokens": 203038.0,
+      "step": 130
+    },
+    {
+      "epoch": 0.25794564716720403,
+      "grad_norm": 2.6612443923950195,
+      "learning_rate": 9.16359163591636e-05,
+      "loss": 2.1091,
+      "mean_token_accuracy": 0.5669840469956398,
+      "num_tokens": 219884.0,
+      "step": 140
+    },
+    {
+      "epoch": 0.2763703362505758,
+      "grad_norm": 2.5735461711883545,
+      "learning_rate": 9.102091020910209e-05,
+      "loss": 2.1719,
+      "mean_token_accuracy": 0.5732400707900525,
+      "num_tokens": 234422.0,
+      "step": 150
+    },
+    {
+      "epoch": 0.29479502533394747,
+      "grad_norm": 2.4123075008392334,
+      "learning_rate": 9.040590405904059e-05,
+      "loss": 2.0741,
+      "mean_token_accuracy": 0.5853110387921333,
+      "num_tokens": 252658.0,
+      "step": 160
+    },
+    {
+      "epoch": 0.3132197144173192,
+      "grad_norm": 4.181519508361816,
+      "learning_rate": 8.97908979089791e-05,
+      "loss": 2.0645,
+      "mean_token_accuracy": 0.5933280706405639,
+      "num_tokens": 266598.0,
+      "step": 170
+    },
+    {
+      "epoch": 0.3316444035006909,
+      "grad_norm": 6.70389986038208,
+      "learning_rate": 8.917589175891759e-05,
+      "loss": 2.0246,
+      "mean_token_accuracy": 0.5990256905555725,
+      "num_tokens": 283568.0,
+      "step": 180
+    },
+    {
+      "epoch": 0.35006909258406266,
+      "grad_norm": 3.190056085586548,
+      "learning_rate": 8.85608856088561e-05,
+      "loss": 2.0359,
+      "mean_token_accuracy": 0.5828932531177997,
+      "num_tokens": 299479.0,
+      "step": 190
+    },
+    {
+      "epoch": 0.36849378166743435,
+      "grad_norm": 2.202435255050659,
+      "learning_rate": 8.794587945879459e-05,
+      "loss": 1.9978,
+      "mean_token_accuracy": 0.5923964321613312,
+      "num_tokens": 314808.0,
+      "step": 200
+    },
+    {
+      "epoch": 0.3869184707508061,
+      "grad_norm": 3.021692991256714,
+      "learning_rate": 8.73308733087331e-05,
+      "loss": 1.9762,
+      "mean_token_accuracy": 0.5954224966466427,
+      "num_tokens": 330418.0,
+      "step": 210
+    },
+    {
+      "epoch": 0.4053431598341778,
+      "grad_norm": 2.3011951446533203,
+      "learning_rate": 8.67158671586716e-05,
+      "loss": 1.9468,
+      "mean_token_accuracy": 0.6064199827611446,
+      "num_tokens": 345678.0,
+      "step": 220
+    },
+    {
+      "epoch": 0.42376784891754954,
+      "grad_norm": 3.2118687629699707,
+      "learning_rate": 8.610086100861009e-05,
+      "loss": 2.0322,
+      "mean_token_accuracy": 0.5958029434084893,
+      "num_tokens": 359806.0,
+      "step": 230
+    },
+    {
+      "epoch": 0.44219253800092123,
+      "grad_norm": 3.341315269470215,
+      "learning_rate": 8.548585485854859e-05,
+      "loss": 1.84,
+      "mean_token_accuracy": 0.6245468840003013,
+      "num_tokens": 376486.0,
+      "step": 240
+    },
+    {
+      "epoch": 0.460617227084293,
+      "grad_norm": 3.6315159797668457,
+      "learning_rate": 8.48708487084871e-05,
+      "loss": 1.9745,
+      "mean_token_accuracy": 0.6007553555071354,
+      "num_tokens": 390325.0,
+      "step": 250
+    },
+    {
+      "epoch": 0.47904191616766467,
+      "grad_norm": 1.6510157585144043,
+      "learning_rate": 8.425584255842559e-05,
+      "loss": 1.8946,
+      "mean_token_accuracy": 0.6109437264502049,
+      "num_tokens": 404030.0,
+      "step": 260
+    },
+    {
+      "epoch": 0.49746660525103636,
+      "grad_norm": 1.7643793821334839,
+      "learning_rate": 8.364083640836408e-05,
+      "loss": 1.9053,
+      "mean_token_accuracy": 0.6258256755769253,
+      "num_tokens": 419835.0,
+      "step": 270
+    },
+    {
+      "epoch": 0.5158912943344081,
+      "grad_norm": 3.0207324028015137,
+      "learning_rate": 8.302583025830258e-05,
+      "loss": 1.9201,
+      "mean_token_accuracy": 0.6229945972561837,
+      "num_tokens": 434982.0,
+      "step": 280
+    },
+    {
+      "epoch": 0.5343159834177799,
+      "grad_norm": 1.9562475681304932,
+      "learning_rate": 8.241082410824109e-05,
+      "loss": 1.792,
+      "mean_token_accuracy": 0.6339600890874862,
+      "num_tokens": 451471.0,
+      "step": 290
+    },
+    {
+      "epoch": 0.5527406725011516,
+      "grad_norm": 2.1910619735717773,
+      "learning_rate": 8.17958179581796e-05,
+      "loss": 1.8753,
+      "mean_token_accuracy": 0.6094988606870174,
+      "num_tokens": 468561.0,
+      "step": 300
+    },
+    {
+      "epoch": 0.5711653615845232,
+      "grad_norm": 2.169729471206665,
+      "learning_rate": 8.118081180811809e-05,
+      "loss": 1.8323,
+      "mean_token_accuracy": 0.6192840307950973,
+      "num_tokens": 483236.0,
+      "step": 310
+    },
+    {
+      "epoch": 0.5895900506678949,
+      "grad_norm": 3.562459945678711,
+      "learning_rate": 8.056580565805658e-05,
+      "loss": 1.839,
+      "mean_token_accuracy": 0.6167505770921707,
+      "num_tokens": 498223.0,
+      "step": 320
+    },
+    {
+      "epoch": 0.6080147397512667,
+      "grad_norm": 2.991919994354248,
+      "learning_rate": 7.995079950799509e-05,
+      "loss": 1.8818,
+      "mean_token_accuracy": 0.6214108224958181,
+      "num_tokens": 514745.0,
+      "step": 330
+    },
+    {
+      "epoch": 0.6264394288346384,
+      "grad_norm": 1.9438029527664185,
+      "learning_rate": 7.933579335793359e-05,
+      "loss": 1.6811,
+      "mean_token_accuracy": 0.6411525875329971,
+      "num_tokens": 529385.0,
+      "step": 340
+    },
+    {
+      "epoch": 0.6448641179180101,
+      "grad_norm": 2.60821795463562,
+      "learning_rate": 7.872078720787208e-05,
+      "loss": 1.6387,
+      "mean_token_accuracy": 0.6569411031901836,
+      "num_tokens": 544502.0,
+      "step": 350
+    },
+    {
+      "epoch": 0.6632888070013818,
+      "grad_norm": 2.1682169437408447,
+      "learning_rate": 7.810578105781058e-05,
+      "loss": 1.8183,
+      "mean_token_accuracy": 0.6362691901624202,
+      "num_tokens": 560484.0,
+      "step": 360
+    },
+    {
+      "epoch": 0.6817134960847536,
+      "grad_norm": 2.3680660724639893,
+      "learning_rate": 7.749077490774908e-05,
+      "loss": 1.7239,
+      "mean_token_accuracy": 0.6350000731647014,
+      "num_tokens": 575806.0,
+      "step": 370
+    },
+    {
+      "epoch": 0.7001381851681253,
+      "grad_norm": 3.8596909046173096,
+      "learning_rate": 7.687576875768758e-05,
+      "loss": 1.7321,
+      "mean_token_accuracy": 0.6481119975447655,
+      "num_tokens": 591583.0,
+      "step": 380
+    },
+    {
+      "epoch": 0.718562874251497,
+      "grad_norm": 2.055788040161133,
+      "learning_rate": 7.626076260762607e-05,
+      "loss": 1.8335,
+      "mean_token_accuracy": 0.6327963292598724,
+      "num_tokens": 605048.0,
+      "step": 390
+    },
+    {
+      "epoch": 0.7369875633348687,
+      "grad_norm": 1.443811297416687,
+      "learning_rate": 7.564575645756458e-05,
+      "loss": 1.6904,
+      "mean_token_accuracy": 0.6381124444305897,
+      "num_tokens": 620806.0,
+      "step": 400
+    },
+    {
+      "epoch": 0.7554122524182404,
+      "grad_norm": 2.873528242111206,
+      "learning_rate": 7.503075030750309e-05,
+      "loss": 1.8429,
+      "mean_token_accuracy": 0.6189649045467377,
+      "num_tokens": 635987.0,
+      "step": 410
+    },
+    {
+      "epoch": 0.7738369415016122,
+      "grad_norm": 2.4668147563934326,
+      "learning_rate": 7.441574415744159e-05,
+      "loss": 1.7066,
+      "mean_token_accuracy": 0.644946389645338,
+      "num_tokens": 653051.0,
+      "step": 420
+    },
+    {
+      "epoch": 0.7922616305849839,
+      "grad_norm": 2.8157310485839844,
+      "learning_rate": 7.380073800738008e-05,
+      "loss": 1.6648,
+      "mean_token_accuracy": 0.6482915267348289,
+      "num_tokens": 666431.0,
+      "step": 430
+    },
+    {
+      "epoch": 0.8106863196683556,
+      "grad_norm": 4.145294666290283,
+      "learning_rate": 7.318573185731857e-05,
+      "loss": 1.6854,
+      "mean_token_accuracy": 0.657547478377819,
+      "num_tokens": 681833.0,
+      "step": 440
+    },
+    {
+      "epoch": 0.8291110087517273,
+      "grad_norm": 3.8063929080963135,
+      "learning_rate": 7.257072570725708e-05,
+      "loss": 1.7735,
+      "mean_token_accuracy": 0.6324092581868171,
+      "num_tokens": 699060.0,
+      "step": 450
+    },
+    {
+      "epoch": 0.8475356978350991,
+      "grad_norm": 1.7125282287597656,
+      "learning_rate": 7.195571955719558e-05,
+      "loss": 1.77,
+      "mean_token_accuracy": 0.6430839493870735,
+      "num_tokens": 715589.0,
+      "step": 460
+    },
+    {
+      "epoch": 0.8659603869184708,
+      "grad_norm": 1.7735799551010132,
+      "learning_rate": 7.134071340713407e-05,
+      "loss": 1.7086,
+      "mean_token_accuracy": 0.6416696727275848,
+      "num_tokens": 730089.0,
+      "step": 470
+    },
+    {
+      "epoch": 0.8843850760018425,
+      "grad_norm": 1.709375262260437,
+      "learning_rate": 7.072570725707257e-05,
+      "loss": 1.6076,
+      "mean_token_accuracy": 0.6620715044438839,
+      "num_tokens": 746328.0,
+      "step": 480
+    },
+    {
+      "epoch": 0.9028097650852142,
+      "grad_norm": 1.7753926515579224,
+      "learning_rate": 7.011070110701108e-05,
+      "loss": 1.775,
+      "mean_token_accuracy": 0.6423866167664528,
+      "num_tokens": 761359.0,
+      "step": 490
+    },
+    {
+      "epoch": 0.921234454168586,
+      "grad_norm": 2.611227512359619,
+      "learning_rate": 6.949569495694957e-05,
+      "loss": 1.7573,
+      "mean_token_accuracy": 0.6375408962368965,
+      "num_tokens": 776148.0,
+      "step": 500
+    },
+    {
+      "epoch": 0.9396591432519577,
+      "grad_norm": 3.674412965774536,
+      "learning_rate": 6.888068880688808e-05,
+      "loss": 1.6506,
+      "mean_token_accuracy": 0.660132697224617,
+      "num_tokens": 793684.0,
+      "step": 510
+    },
+    {
+      "epoch": 0.9580838323353293,
+      "grad_norm": 1.4588079452514648,
+      "learning_rate": 6.826568265682657e-05,
+      "loss": 1.584,
+      "mean_token_accuracy": 0.6719754695892334,
+      "num_tokens": 810075.0,
+      "step": 520
+    },
+    {
+      "epoch": 0.976508521418701,
+      "grad_norm": 2.6270811557769775,
+      "learning_rate": 6.765067650676508e-05,
+      "loss": 1.8693,
+      "mean_token_accuracy": 0.6431014843285083,
+      "num_tokens": 821845.0,
+      "step": 530
+    },
+    {
+      "epoch": 0.9949332105020727,
+      "grad_norm": 3.1522443294525146,
+      "learning_rate": 6.703567035670358e-05,
+      "loss": 1.616,
+      "mean_token_accuracy": 0.6511368542909622,
+      "num_tokens": 837314.0,
+      "step": 540
+    },
+    {
+      "epoch": 1.0128972823583602,
+      "grad_norm": 1.6306216716766357,
+      "learning_rate": 6.642066420664207e-05,
+      "loss": 1.879,
+      "mean_token_accuracy": 0.6300988514453937,
+      "num_tokens": 850588.0,
+      "step": 550
+    },
+    {
+      "epoch": 1.0313219714417319,
+      "grad_norm": 3.6696395874023438,
+      "learning_rate": 6.580565805658057e-05,
+      "loss": 1.6574,
+      "mean_token_accuracy": 0.6459669299423695,
+      "num_tokens": 864714.0,
+      "step": 560
+    },
+    {
+      "epoch": 1.0497466605251036,
+      "grad_norm": 2.42344331741333,
+      "learning_rate": 6.519065190651907e-05,
+      "loss": 1.6792,
+      "mean_token_accuracy": 0.6428927913308143,
+      "num_tokens": 880970.0,
+      "step": 570
+    },
+    {
+      "epoch": 1.0681713496084753,
+      "grad_norm": 2.758784770965576,
+      "learning_rate": 6.457564575645757e-05,
+      "loss": 1.6636,
+      "mean_token_accuracy": 0.6472427472472191,
+      "num_tokens": 894346.0,
+      "step": 580
+    },
+    {
+      "epoch": 1.0865960386918472,
+      "grad_norm": 3.1409521102905273,
+      "learning_rate": 6.396063960639606e-05,
+      "loss": 1.6937,
+      "mean_token_accuracy": 0.6384285531938076,
+      "num_tokens": 913076.0,
+      "step": 590
+    },
+    {
+      "epoch": 1.1050207277752189,
+      "grad_norm": 2.055471658706665,
+      "learning_rate": 6.334563345633456e-05,
+      "loss": 1.6229,
+      "mean_token_accuracy": 0.6535055659711361,
+      "num_tokens": 927828.0,
+      "step": 600
+    },
+    {
+      "epoch": 1.1234454168585906,
+      "grad_norm": 1.4776051044464111,
+      "learning_rate": 6.273062730627307e-05,
+      "loss": 1.5533,
+      "mean_token_accuracy": 0.6595104590058327,
+      "num_tokens": 941252.0,
+      "step": 610
+    },
+    {
+      "epoch": 1.1418701059419623,
+      "grad_norm": 2.9976112842559814,
+      "learning_rate": 6.211562115621157e-05,
+      "loss": 1.5572,
+      "mean_token_accuracy": 0.6735355719923973,
+      "num_tokens": 957399.0,
+      "step": 620
+    },
+    {
+      "epoch": 1.160294795025334,
+      "grad_norm": 2.034322738647461,
+      "learning_rate": 6.150061500615007e-05,
+      "loss": 1.7031,
+      "mean_token_accuracy": 0.659538009762764,
+      "num_tokens": 973536.0,
+      "step": 630
+    },
+    {
+      "epoch": 1.1787194841087056,
+      "grad_norm": 2.450007438659668,
+      "learning_rate": 6.0885608856088563e-05,
+      "loss": 1.8057,
+      "mean_token_accuracy": 0.6286990121006966,
+      "num_tokens": 987842.0,
+      "step": 640
+    },
+    {
+      "epoch": 1.1971441731920773,
+      "grad_norm": 1.9177062511444092,
+      "learning_rate": 6.027060270602707e-05,
+      "loss": 1.7082,
+      "mean_token_accuracy": 0.6420982472598553,
+      "num_tokens": 1003454.0,
+      "step": 650
+    },
+    {
+      "epoch": 1.215568862275449,
+      "grad_norm": 2.712975263595581,
+      "learning_rate": 5.9655596555965566e-05,
+      "loss": 1.5976,
+      "mean_token_accuracy": 0.651873255521059,
+      "num_tokens": 1017354.0,
+      "step": 660
+    },
+    {
+      "epoch": 1.2339935513588207,
+      "grad_norm": 2.0437071323394775,
+      "learning_rate": 5.904059040590406e-05,
+      "loss": 1.654,
+      "mean_token_accuracy": 0.6428793512284756,
+      "num_tokens": 1031349.0,
+      "step": 670
+    },
+    {
+      "epoch": 1.2524182404421924,
+      "grad_norm": 4.048579216003418,
+      "learning_rate": 5.8425584255842556e-05,
+      "loss": 1.6849,
+      "mean_token_accuracy": 0.6524994812905789,
+      "num_tokens": 1046888.0,
+      "step": 680
+    },
+    {
+      "epoch": 1.2708429295255643,
+      "grad_norm": 1.9990712404251099,
+      "learning_rate": 5.7810578105781064e-05,
+      "loss": 1.5438,
+      "mean_token_accuracy": 0.6665605463087558,
+      "num_tokens": 1063677.0,
+      "step": 690
+    },
+    {
+      "epoch": 1.289267618608936,
+      "grad_norm": 4.054327487945557,
+      "learning_rate": 5.7195571955719566e-05,
+      "loss": 1.7604,
+      "mean_token_accuracy": 0.6362201184034347,
+      "num_tokens": 1078450.0,
+      "step": 700
+    },
+    {
+      "epoch": 1.3076923076923077,
+      "grad_norm": 1.500022530555725,
+      "learning_rate": 5.658056580565806e-05,
+      "loss": 1.6035,
+      "mean_token_accuracy": 0.6589202269911766,
+      "num_tokens": 1095184.0,
+      "step": 710
+    },
+    {
+      "epoch": 1.3261169967756794,
+      "grad_norm": 1.5381805896759033,
+      "learning_rate": 5.5965559655596555e-05,
+      "loss": 1.4785,
+      "mean_token_accuracy": 0.6745613172650338,
+      "num_tokens": 1110018.0,
+      "step": 720
+    },
+    {
+      "epoch": 1.344541685859051,
+      "grad_norm": 2.767632007598877,
+      "learning_rate": 5.535055350553506e-05,
+      "loss": 1.7756,
+      "mean_token_accuracy": 0.6358517713844776,
+      "num_tokens": 1123395.0,
+      "step": 730
+    },
+    {
+      "epoch": 1.3629663749424228,
+      "grad_norm": 2.790994882583618,
+      "learning_rate": 5.473554735547356e-05,
+      "loss": 1.8279,
+      "mean_token_accuracy": 0.6309185773134232,
+      "num_tokens": 1137612.0,
+      "step": 740
+    },
+    {
+      "epoch": 1.3813910640257947,
+      "grad_norm": 2.408815860748291,
+      "learning_rate": 5.412054120541206e-05,
+      "loss": 1.6681,
+      "mean_token_accuracy": 0.6524201087653637,
+      "num_tokens": 1153122.0,
+      "step": 750
+    },
+    {
+      "epoch": 1.3998157531091664,
+      "grad_norm": 1.718011736869812,
+      "learning_rate": 5.3505535055350554e-05,
+      "loss": 1.5317,
+      "mean_token_accuracy": 0.6702338352799415,
+      "num_tokens": 1170060.0,
+      "step": 760
+    },
+    {
+      "epoch": 1.418240442192538,
+      "grad_norm": 3.807502508163452,
+      "learning_rate": 5.289052890528906e-05,
+      "loss": 1.6218,
+      "mean_token_accuracy": 0.6621288940310478,
+      "num_tokens": 1184871.0,
+      "step": 770
+    },
+    {
+      "epoch": 1.4366651312759098,
+      "grad_norm": 1.927329182624817,
+      "learning_rate": 5.227552275522756e-05,
+      "loss": 1.5677,
+      "mean_token_accuracy": 0.6642054319381714,
+      "num_tokens": 1199929.0,
+      "step": 780
+    },
+    {
+      "epoch": 1.4550898203592815,
+      "grad_norm": 1.845070481300354,
+      "learning_rate": 5.166051660516605e-05,
+      "loss": 1.6204,
+      "mean_token_accuracy": 0.6633838757872581,
+      "num_tokens": 1215540.0,
+      "step": 790
+    },
+    {
+      "epoch": 1.4735145094426532,
+      "grad_norm": 2.4076995849609375,
+      "learning_rate": 5.104551045510455e-05,
+      "loss": 1.598,
+      "mean_token_accuracy": 0.6614152923226356,
+      "num_tokens": 1232591.0,
+      "step": 800
+    },
+    {
+      "epoch": 1.4919391985260249,
+      "grad_norm": 3.6199724674224854,
+      "learning_rate": 5.0430504305043055e-05,
+      "loss": 1.5614,
+      "mean_token_accuracy": 0.664795532822609,
+      "num_tokens": 1247973.0,
+      "step": 810
+    },
+    {
+      "epoch": 1.5103638876093965,
+      "grad_norm": 2.4386658668518066,
+      "learning_rate": 4.9815498154981556e-05,
+      "loss": 1.6204,
+      "mean_token_accuracy": 0.656517218053341,
+      "num_tokens": 1264392.0,
+      "step": 820
+    },
+    {
+      "epoch": 1.5287885766927682,
+      "grad_norm": 1.7179166078567505,
+      "learning_rate": 4.920049200492005e-05,
+      "loss": 1.667,
+      "mean_token_accuracy": 0.6481011256575584,
+      "num_tokens": 1281176.0,
+      "step": 830
+    },
+    {
+      "epoch": 1.54721326577614,
+      "grad_norm": 1.5895161628723145,
+      "learning_rate": 4.858548585485855e-05,
+      "loss": 1.7653,
+      "mean_token_accuracy": 0.6350654922425747,
+      "num_tokens": 1295438.0,
+      "step": 840
+    },
+    {
+      "epoch": 1.5656379548595116,
+      "grad_norm": 2.192767381668091,
+      "learning_rate": 4.797047970479705e-05,
+      "loss": 1.7234,
+      "mean_token_accuracy": 0.6520352639257908,
+      "num_tokens": 1311458.0,
+      "step": 850
+    },
+    {
+      "epoch": 1.5840626439428833,
+      "grad_norm": 2.573124885559082,
+      "learning_rate": 4.7355473554735555e-05,
+      "loss": 1.6244,
+      "mean_token_accuracy": 0.6589855045080185,
+      "num_tokens": 1326482.0,
+      "step": 860
+    },
+    {
+      "epoch": 1.6024873330262552,
+      "grad_norm": 3.809417247772217,
+      "learning_rate": 4.674046740467405e-05,
+      "loss": 1.6239,
+      "mean_token_accuracy": 0.6600351750850677,
+      "num_tokens": 1341909.0,
+      "step": 870
+    },
+    {
+      "epoch": 1.620912022109627,
+      "grad_norm": 1.531394124031067,
+      "learning_rate": 4.612546125461255e-05,
+      "loss": 1.5705,
+      "mean_token_accuracy": 0.6624191544950009,
+      "num_tokens": 1357812.0,
+      "step": 880
+    },
+    {
+      "epoch": 1.6393367111929986,
+      "grad_norm": 3.8407554626464844,
+      "learning_rate": 4.5510455104551046e-05,
+      "loss": 1.6656,
+      "mean_token_accuracy": 0.6529517590999603,
+      "num_tokens": 1372898.0,
+      "step": 890
+    },
+    {
+      "epoch": 1.6577614002763703,
+      "grad_norm": 2.4914557933807373,
+      "learning_rate": 4.489544895448955e-05,
+      "loss": 1.5695,
+      "mean_token_accuracy": 0.675298410654068,
+      "num_tokens": 1387354.0,
+      "step": 900
+    },
+    {
+      "epoch": 1.676186089359742,
+      "grad_norm": 2.8756096363067627,
+      "learning_rate": 4.428044280442805e-05,
+      "loss": 1.6854,
+      "mean_token_accuracy": 0.64402234852314,
+      "num_tokens": 1402343.0,
+      "step": 910
+    },
+    {
+      "epoch": 1.694610778443114,
+      "grad_norm": 1.9504696130752563,
+      "learning_rate": 4.366543665436655e-05,
+      "loss": 1.6059,
+      "mean_token_accuracy": 0.6611929297447204,
+      "num_tokens": 1418248.0,
+      "step": 920
+    },
+    {
+      "epoch": 1.7130354675264856,
+      "grad_norm": 1.6919995546340942,
+      "learning_rate": 4.3050430504305045e-05,
+      "loss": 1.5615,
+      "mean_token_accuracy": 0.6665925502777099,
+      "num_tokens": 1432999.0,
+      "step": 930
+    },
+    {
+      "epoch": 1.7314601566098573,
+      "grad_norm": 2.0128772258758545,
+      "learning_rate": 4.243542435424355e-05,
+      "loss": 1.5776,
+      "mean_token_accuracy": 0.6651003785431385,
+      "num_tokens": 1449366.0,
+      "step": 940
+    },
+    {
+      "epoch": 1.749884845693229,
+      "grad_norm": 2.1715657711029053,
+      "learning_rate": 4.182041820418204e-05,
+      "loss": 1.6152,
+      "mean_token_accuracy": 0.6561852261424065,
+      "num_tokens": 1465702.0,
+      "step": 950
+    },
+    {
+      "epoch": 1.7683095347766007,
+      "grad_norm": 2.274376153945923,
+      "learning_rate": 4.120541205412054e-05,
+      "loss": 1.4871,
+      "mean_token_accuracy": 0.6793628737330437,
+      "num_tokens": 1480603.0,
+      "step": 960
+    },
+    {
+      "epoch": 1.7867342238599724,
+      "grad_norm": 3.4055070877075195,
+      "learning_rate": 4.0590405904059045e-05,
+      "loss": 1.5607,
+      "mean_token_accuracy": 0.6566946871578694,
+      "num_tokens": 1495993.0,
+      "step": 970
+    },
+    {
+      "epoch": 1.805158912943344,
+      "grad_norm": 3.325863838195801,
+      "learning_rate": 3.9975399753997546e-05,
+      "loss": 1.6572,
+      "mean_token_accuracy": 0.6670118771493435,
+      "num_tokens": 1512422.0,
+      "step": 980
+    },
+    {
+      "epoch": 1.8235836020267158,
+      "grad_norm": 2.7312819957733154,
+      "learning_rate": 3.936039360393604e-05,
+      "loss": 1.6001,
+      "mean_token_accuracy": 0.6647168889641761,
+      "num_tokens": 1528258.0,
+      "step": 990
+    },
+    {
+      "epoch": 1.8420082911100875,
+      "grad_norm": 1.7766013145446777,
+      "learning_rate": 3.874538745387454e-05,
+      "loss": 1.5707,
+      "mean_token_accuracy": 0.6695287272334098,
+      "num_tokens": 1544350.0,
+      "step": 1000
+    },
+    {
+      "epoch": 1.8604329801934592,
+      "grad_norm": 1.8994312286376953,
+      "learning_rate": 3.813038130381304e-05,
+      "loss": 1.613,
+      "mean_token_accuracy": 0.6718984022736549,
+      "num_tokens": 1558824.0,
+      "step": 1010
+    },
+    {
+      "epoch": 1.8788576692768308,
+      "grad_norm": 1.7150542736053467,
+      "learning_rate": 3.7515375153751545e-05,
+      "loss": 1.5584,
+      "mean_token_accuracy": 0.6699837028980256,
+      "num_tokens": 1576690.0,
+      "step": 1020
+    },
+    {
+      "epoch": 1.8972823583602025,
+      "grad_norm": 2.059967279434204,
+      "learning_rate": 3.690036900369004e-05,
+      "loss": 1.5869,
+      "mean_token_accuracy": 0.6748946160078049,
+      "num_tokens": 1592835.0,
+      "step": 1030
+    },
+    {
+      "epoch": 1.9157070474435742,
+      "grad_norm": 1.863437294960022,
+      "learning_rate": 3.628536285362854e-05,
+      "loss": 1.5838,
+      "mean_token_accuracy": 0.655283273011446,
+      "num_tokens": 1609698.0,
+      "step": 1040
+    },
+    {
+      "epoch": 1.9341317365269461,
+      "grad_norm": 3.684446096420288,
+      "learning_rate": 3.5670356703567036e-05,
+      "loss": 1.5472,
+      "mean_token_accuracy": 0.6692177668213845,
+      "num_tokens": 1625756.0,
+      "step": 1050
+    },
+    {
+      "epoch": 1.9525564256103178,
+      "grad_norm": 4.181140899658203,
+      "learning_rate": 3.505535055350554e-05,
+      "loss": 1.4674,
+      "mean_token_accuracy": 0.6852930411696434,
+      "num_tokens": 1643829.0,
+      "step": 1060
+    },
+    {
+      "epoch": 1.9709811146936895,
+      "grad_norm": 2.2308173179626465,
+      "learning_rate": 3.444034440344404e-05,
+      "loss": 1.6001,
+      "mean_token_accuracy": 0.6600222229957581,
+      "num_tokens": 1656497.0,
+      "step": 1070
+    },
+    {
+      "epoch": 1.9894058037770612,
+      "grad_norm": 1.6924246549606323,
+      "learning_rate": 3.382533825338254e-05,
+      "loss": 1.3854,
+      "mean_token_accuracy": 0.6935430377721786,
+      "num_tokens": 1673101.0,
+      "step": 1080
+    },
+    {
+      "epoch": 2.0073698756333487,
+      "grad_norm": 2.8476662635803223,
+      "learning_rate": 3.3210332103321035e-05,
+      "loss": 1.6707,
+      "mean_token_accuracy": 0.6428498587547204,
+      "num_tokens": 1688433.0,
+      "step": 1090
+    },
+    {
+      "epoch": 2.0257945647167204,
+      "grad_norm": 2.2436559200286865,
+      "learning_rate": 3.259532595325954e-05,
+      "loss": 1.5851,
+      "mean_token_accuracy": 0.6709273040294648,
+      "num_tokens": 1706958.0,
+      "step": 1100
+    },
+    {
+      "epoch": 2.044219253800092,
+      "grad_norm": 1.4724531173706055,
+      "learning_rate": 3.198031980319803e-05,
+      "loss": 1.5028,
+      "mean_token_accuracy": 0.6706318959593773,
+      "num_tokens": 1722381.0,
+      "step": 1110
+    },
+    {
+      "epoch": 2.0626439428834638,
+      "grad_norm": 1.953519582748413,
+      "learning_rate": 3.136531365313653e-05,
+      "loss": 1.5389,
+      "mean_token_accuracy": 0.6631861045956612,
+      "num_tokens": 1740659.0,
+      "step": 1120
+    },
+    {
+      "epoch": 2.0810686319668354,
+      "grad_norm": 2.0097978115081787,
+      "learning_rate": 3.0750307503075034e-05,
+      "loss": 1.5499,
+      "mean_token_accuracy": 0.6736045613884926,
+      "num_tokens": 1756611.0,
+      "step": 1130
+    },
+    {
+      "epoch": 2.099493321050207,
+      "grad_norm": 1.5125641822814941,
+      "learning_rate": 3.0135301353013536e-05,
+      "loss": 1.5976,
+      "mean_token_accuracy": 0.6485340878367424,
+      "num_tokens": 1772542.0,
+      "step": 1140
+    },
+    {
+      "epoch": 2.117918010133579,
+      "grad_norm": 3.323991537094116,
+      "learning_rate": 2.952029520295203e-05,
+      "loss": 1.657,
+      "mean_token_accuracy": 0.6505417212843895,
+      "num_tokens": 1788151.0,
+      "step": 1150
+    },
+    {
+      "epoch": 2.1363426992169505,
+      "grad_norm": 1.968610405921936,
+      "learning_rate": 2.8905289052890532e-05,
+      "loss": 1.493,
+      "mean_token_accuracy": 0.6852642297744751,
+      "num_tokens": 1804815.0,
+      "step": 1160
+    },
+    {
+      "epoch": 2.154767388300322,
+      "grad_norm": 3.373903512954712,
+      "learning_rate": 2.829028290282903e-05,
+      "loss": 1.5195,
+      "mean_token_accuracy": 0.6652759924530983,
+      "num_tokens": 1818226.0,
+      "step": 1170
+    },
+    {
+      "epoch": 2.1731920773836944,
+      "grad_norm": 2.232060432434082,
+      "learning_rate": 2.767527675276753e-05,
+      "loss": 1.5121,
+      "mean_token_accuracy": 0.6762837044894695,
+      "num_tokens": 1833400.0,
+      "step": 1180
+    },
+    {
+      "epoch": 2.191616766467066,
+      "grad_norm": 1.888635516166687,
+      "learning_rate": 2.706027060270603e-05,
+      "loss": 1.6081,
+      "mean_token_accuracy": 0.6562661565840244,
+      "num_tokens": 1847505.0,
+      "step": 1190
+    },
+    {
+      "epoch": 2.2100414555504377,
+      "grad_norm": 2.28125262260437,
+      "learning_rate": 2.650676506765068e-05,
+      "loss": 1.5981,
+      "mean_token_accuracy": 0.6534711696207524,
+      "num_tokens": 1862044.0,
+      "step": 1200
+    },
+    {
+      "epoch": 2.2284661446338094,
+      "grad_norm": 2.1634366512298584,
+      "learning_rate": 2.5891758917589176e-05,
+      "loss": 1.6522,
+      "mean_token_accuracy": 0.6541110493242741,
+      "num_tokens": 1876889.0,
+      "step": 1210
+    },
+    {
+      "epoch": 2.246890833717181,
+      "grad_norm": 2.3856070041656494,
+      "learning_rate": 2.5276752767527677e-05,
+      "loss": 1.5706,
+      "mean_token_accuracy": 0.6649116739630699,
+      "num_tokens": 1893388.0,
+      "step": 1220
+    },
+    {
+      "epoch": 2.265315522800553,
+      "grad_norm": 2.3563289642333984,
+      "learning_rate": 2.4661746617466175e-05,
+      "loss": 1.6115,
+      "mean_token_accuracy": 0.6710579909384251,
+      "num_tokens": 1908415.0,
+      "step": 1230
+    },
+    {
+      "epoch": 2.2837402118839245,
+      "grad_norm": 1.7748916149139404,
+      "learning_rate": 2.4046740467404673e-05,
+      "loss": 1.5964,
+      "mean_token_accuracy": 0.661432110518217,
+      "num_tokens": 1922173.0,
+      "step": 1240
+    },
+    {
+      "epoch": 2.302164900967296,
+      "grad_norm": 3.3364036083221436,
+      "learning_rate": 2.3431734317343175e-05,
+      "loss": 1.5898,
+      "mean_token_accuracy": 0.6684726983308792,
+      "num_tokens": 1936339.0,
+      "step": 1250
+    },
+    {
+      "epoch": 2.320589590050668,
+      "grad_norm": 1.6432230472564697,
+      "learning_rate": 2.2816728167281673e-05,
+      "loss": 1.7973,
+      "mean_token_accuracy": 0.6203286804258823,
+      "num_tokens": 1950870.0,
+      "step": 1260
+    },
+    {
+      "epoch": 2.3390142791340396,
+      "grad_norm": 1.678228497505188,
+      "learning_rate": 2.220172201722017e-05,
+      "loss": 1.4057,
+      "mean_token_accuracy": 0.6923253484070301,
+      "num_tokens": 1967394.0,
+      "step": 1270
+    },
+    {
+      "epoch": 2.3574389682174113,
+      "grad_norm": 2.503005266189575,
+      "learning_rate": 2.1586715867158673e-05,
+      "loss": 1.643,
+      "mean_token_accuracy": 0.6646282590925694,
+      "num_tokens": 1983865.0,
+      "step": 1280
+    },
+    {
+      "epoch": 2.375863657300783,
+      "grad_norm": 2.67020583152771,
+      "learning_rate": 2.097170971709717e-05,
+      "loss": 1.3788,
+      "mean_token_accuracy": 0.6916278369724751,
+      "num_tokens": 2000344.0,
+      "step": 1290
+    },
+    {
+      "epoch": 2.3942883463841547,
+      "grad_norm": 2.9962821006774902,
+      "learning_rate": 2.035670356703567e-05,
+      "loss": 1.5506,
+      "mean_token_accuracy": 0.6763791911303997,
+      "num_tokens": 2014336.0,
+      "step": 1300
+    },
+    {
+      "epoch": 2.4127130354675264,
+      "grad_norm": 1.9718595743179321,
+      "learning_rate": 1.974169741697417e-05,
+      "loss": 1.422,
+      "mean_token_accuracy": 0.6856225565075874,
+      "num_tokens": 2027788.0,
+      "step": 1310
+    },
+    {
+      "epoch": 2.431137724550898,
+      "grad_norm": 2.0590851306915283,
+      "learning_rate": 1.912669126691267e-05,
+      "loss": 1.5242,
+      "mean_token_accuracy": 0.6755510538816452,
+      "num_tokens": 2042151.0,
+      "step": 1320
+    },
+    {
+      "epoch": 2.4495624136342697,
+      "grad_norm": 3.358384609222412,
+      "learning_rate": 1.851168511685117e-05,
+      "loss": 1.7132,
+      "mean_token_accuracy": 0.6465204678475857,
+      "num_tokens": 2055971.0,
+      "step": 1330
+    },
+    {
+      "epoch": 2.4679871027176414,
+      "grad_norm": 2.9107906818389893,
+      "learning_rate": 1.7896678966789668e-05,
+      "loss": 1.515,
+      "mean_token_accuracy": 0.681264215707779,
+      "num_tokens": 2072566.0,
+      "step": 1340
+    },
+    {
+      "epoch": 2.4864117918010136,
+      "grad_norm": 1.770975947380066,
+      "learning_rate": 1.7281672816728166e-05,
+      "loss": 1.5032,
+      "mean_token_accuracy": 0.6722722187638283,
+      "num_tokens": 2088145.0,
+      "step": 1350
+    },
+    {
+      "epoch": 2.504836480884385,
+      "grad_norm": 1.7713598012924194,
+      "learning_rate": 1.6666666666666667e-05,
+      "loss": 1.5416,
+      "mean_token_accuracy": 0.6729029573500156,
+      "num_tokens": 2106542.0,
+      "step": 1360
+    },
+    {
+      "epoch": 2.523261169967757,
+      "grad_norm": 2.127868890762329,
+      "learning_rate": 1.6051660516605166e-05,
+      "loss": 1.5215,
+      "mean_token_accuracy": 0.6672196105122566,
+      "num_tokens": 2123560.0,
+      "step": 1370
+    },
+    {
+      "epoch": 2.5416858590511286,
+      "grad_norm": 2.1167984008789062,
+      "learning_rate": 1.5436654366543664e-05,
+      "loss": 1.4684,
+      "mean_token_accuracy": 0.6837770715355873,
+      "num_tokens": 2137495.0,
+      "step": 1380
+    },
+    {
+      "epoch": 2.5601105481345003,
+      "grad_norm": 2.207040548324585,
+      "learning_rate": 1.4821648216482165e-05,
+      "loss": 1.5336,
+      "mean_token_accuracy": 0.6734577745199204,
+      "num_tokens": 2151938.0,
+      "step": 1390
+    },
+    {
+      "epoch": 2.578535237217872,
+      "grad_norm": 1.7322977781295776,
+      "learning_rate": 1.4206642066420663e-05,
+      "loss": 1.4342,
+      "mean_token_accuracy": 0.6726964369416237,
+      "num_tokens": 2170151.0,
+      "step": 1400
+    },
+    {
+      "epoch": 2.5969599263012437,
+      "grad_norm": 1.3172131776809692,
+      "learning_rate": 1.3591635916359163e-05,
+      "loss": 1.4978,
+      "mean_token_accuracy": 0.6762748882174492,
+      "num_tokens": 2188585.0,
+      "step": 1410
+    },
+    {
+      "epoch": 2.6153846153846154,
+      "grad_norm": 1.8074936866760254,
+      "learning_rate": 1.2976629766297663e-05,
+      "loss": 1.5174,
+      "mean_token_accuracy": 0.6642862960696221,
+      "num_tokens": 2203375.0,
+      "step": 1420
+    },
+    {
+      "epoch": 2.633809304467987,
+      "grad_norm": 1.885619044303894,
+      "learning_rate": 1.2361623616236164e-05,
+      "loss": 1.5948,
+      "mean_token_accuracy": 0.6671996504068375,
+      "num_tokens": 2218469.0,
+      "step": 1430
+    },
+    {
+      "epoch": 2.652233993551359,
+      "grad_norm": 1.6329900026321411,
+      "learning_rate": 1.1746617466174662e-05,
+      "loss": 1.5492,
+      "mean_token_accuracy": 0.6715722292661667,
+      "num_tokens": 2231834.0,
+      "step": 1440
+    },
+    {
+      "epoch": 2.6706586826347305,
+      "grad_norm": 2.1220619678497314,
+      "learning_rate": 1.1131611316113162e-05,
+      "loss": 1.5341,
+      "mean_token_accuracy": 0.6820433601737023,
+      "num_tokens": 2245721.0,
+      "step": 1450
+    },
+    {
+      "epoch": 2.689083371718102,
+      "grad_norm": 1.9964560270309448,
+      "learning_rate": 1.0516605166051662e-05,
+      "loss": 1.5166,
+      "mean_token_accuracy": 0.6628463111817837,
+      "num_tokens": 2261815.0,
+      "step": 1460
+    },
+    {
+      "epoch": 2.707508060801474,
+      "grad_norm": 12.387279510498047,
+      "learning_rate": 9.90159901599016e-06,
+      "loss": 1.5471,
+      "mean_token_accuracy": 0.6632601089775563,
+      "num_tokens": 2276667.0,
+      "step": 1470
+    },
+    {
+      "epoch": 2.7259327498848456,
+      "grad_norm": 2.0253548622131348,
+      "learning_rate": 9.28659286592866e-06,
+      "loss": 1.5658,
+      "mean_token_accuracy": 0.6789613053202629,
+      "num_tokens": 2291291.0,
+      "step": 1480
+    },
+    {
+      "epoch": 2.7443574389682173,
+      "grad_norm": 1.9108439683914185,
+      "learning_rate": 8.67158671586716e-06,
+      "loss": 1.4854,
+      "mean_token_accuracy": 0.6836533501744271,
+      "num_tokens": 2303999.0,
+      "step": 1490
+    },
+    {
+      "epoch": 2.7627821280515894,
+      "grad_norm": 1.7962969541549683,
+      "learning_rate": 8.05658056580566e-06,
+      "loss": 1.5273,
+      "mean_token_accuracy": 0.6722525358200073,
+      "num_tokens": 2319105.0,
+      "step": 1500
+    },
+    {
+      "epoch": 2.7812068171349607,
+      "grad_norm": 2.1593410968780518,
+      "learning_rate": 7.441574415744158e-06,
+      "loss": 1.5067,
+      "mean_token_accuracy": 0.6785383746027946,
+      "num_tokens": 2334926.0,
+      "step": 1510
+    },
+    {
+      "epoch": 2.799631506218333,
+      "grad_norm": 2.421997308731079,
+      "learning_rate": 6.826568265682657e-06,
+      "loss": 1.6342,
+      "mean_token_accuracy": 0.6506063111126423,
+      "num_tokens": 2349539.0,
+      "step": 1520
+    },
+    {
+      "epoch": 2.818056195301704,
+      "grad_norm": 3.4870669841766357,
+      "learning_rate": 6.211562115621156e-06,
+      "loss": 1.5042,
+      "mean_token_accuracy": 0.6885576844215393,
+      "num_tokens": 2365100.0,
+      "step": 1530
+    },
+    {
+      "epoch": 2.836480884385076,
+      "grad_norm": 3.078742504119873,
+      "learning_rate": 5.596555965559656e-06,
+      "loss": 1.5695,
+      "mean_token_accuracy": 0.6784139782190323,
+      "num_tokens": 2380593.0,
+      "step": 1540
+    },
+    {
+      "epoch": 2.854905573468448,
+      "grad_norm": 1.6315090656280518,
+      "learning_rate": 4.981549815498155e-06,
+      "loss": 1.4953,
+      "mean_token_accuracy": 0.6661784201860428,
+      "num_tokens": 2394706.0,
+      "step": 1550
+    },
+    {
+      "epoch": 2.8733302625518196,
+      "grad_norm": 2.375655174255371,
+      "learning_rate": 4.366543665436655e-06,
+      "loss": 1.4862,
+      "mean_token_accuracy": 0.67956483066082,
+      "num_tokens": 2411550.0,
+      "step": 1560
+    },
+    {
+      "epoch": 2.8917549516351913,
+      "grad_norm": 2.630934476852417,
+      "learning_rate": 3.7515375153751537e-06,
+      "loss": 1.5755,
+      "mean_token_accuracy": 0.6751994095742703,
+      "num_tokens": 2425876.0,
+      "step": 1570
+    },
+    {
+      "epoch": 2.910179640718563,
+      "grad_norm": 3.1176366806030273,
+      "learning_rate": 3.136531365313653e-06,
+      "loss": 1.7321,
+      "mean_token_accuracy": 0.6398597691208124,
+      "num_tokens": 2439891.0,
+      "step": 1580
+    },
+    {
+      "epoch": 2.9286043298019346,
+      "grad_norm": 1.6314257383346558,
+      "learning_rate": 2.5215252152521524e-06,
+      "loss": 1.6167,
+      "mean_token_accuracy": 0.6670875735580921,
+      "num_tokens": 2455716.0,
+      "step": 1590
+    },
+    {
+      "epoch": 2.9470290188853063,
+      "grad_norm": 1.7537063360214233,
+      "learning_rate": 1.9065190651906518e-06,
+      "loss": 1.5896,
+      "mean_token_accuracy": 0.6640482462942601,
+      "num_tokens": 2472772.0,
+      "step": 1600
+    },
+    {
+      "epoch": 2.965453707968678,
+      "grad_norm": 1.577064037322998,
+      "learning_rate": 1.2915129151291513e-06,
+      "loss": 1.4015,
+      "mean_token_accuracy": 0.6940291911363602,
+      "num_tokens": 2490083.0,
+      "step": 1610
+    },
+    {
+      "epoch": 2.9838783970520497,
+      "grad_norm": 1.8871995210647583,
+      "learning_rate": 6.765067650676507e-07,
+      "loss": 1.4477,
+      "mean_token_accuracy": 0.6804400004446507,
+      "num_tokens": 2508552.0,
+      "step": 1620
+    }
+  ],
+  "logging_steps": 10,
+  "max_steps": 1626,
+  "num_input_tokens_seen": 0,
+  "num_train_epochs": 3,
+  "save_steps": 50,
+  "stateful_callbacks": {
+    "TrainerControl": {
+      "args": {
+        "should_epoch_stop": false,
+        "should_evaluate": false,
+        "should_log": false,
+        "should_save": true,
+        "should_training_stop": true
+      },
+      "attributes": {}
+    }
+  },
+  "total_flos": 4.352835556859904e+16,
+  "train_batch_size": 1,
+  "trial_name": null,
+  "trial_params": null
+}

checkpoints/training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:35b9c3a5713cf530404c33da9c6c91e14a277b7b9b8514a70150fcdcbfdfd4ad
+size 5560