Upload folder using huggingface_hub

.gitattributes

@@ -33,3 +33,6 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+checkpoint-3119/tokenizer.json filter=lfs diff=lfs merge=lfs -text
+checkpoint-6237/tokenizer.json filter=lfs diff=lfs merge=lfs -text
+tokenizer.json filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,58 @@
+---
+base_model: google/paligemma2-3b-pt-224
+datasets: mateoguaman/vamos_10pct_gpt5_mini_cocoqa_localized_narratives_fixed
+library_name: transformers
+model_name: google/paligemma2-3b-pt-224
+tags:
+- generated_from_trainer
+- alignment-handbook
+licence: license
+---
+
+# Model Card for google/paligemma2-3b-pt-224
+
+This model is a fine-tuned version of [google/paligemma2-3b-pt-224](https://huggingface.co/google/paligemma2-3b-pt-224) on the [mateoguaman/vamos_10pct_gpt5_mini_cocoqa_localized_narratives_fixed](https://huggingface.co/datasets/mateoguaman/vamos_10pct_gpt5_mini_cocoqa_localized_narratives_fixed) dataset.
+It has been trained using [TRL](https://github.com/huggingface/trl).
+
+## Quick start
+
+```python
+from transformers import pipeline
+
+question = "If you had a time machine, but could only go to the past or the future once and never return, which would you choose and why?"
+generator = pipeline("text-generation", model="None", device="cuda")
+output = generator([{"role": "user", "content": question}], max_new_tokens=128, return_full_text=False)[0]
+print(output["generated_text"])
+```
+
+## Training procedure
+
+[<img src="https://raw.githubusercontent.com/wandb/assets/main/wandb-github-badge-28.svg" alt="Visualize in Weights & Biases" width="150" height="24"/>](https://wandb.ai/tri/paligemma2-3b-pt-224-sft-lora-vamos_10pct_gpt5_mini_cocoqa_localized_narratives_fixed/runs/34idwmc1) 
+
+
+This model was trained with SFT.
+
+### Framework versions
+
+- TRL: 0.15.2
+- Transformers: 4.49.0
+- Pytorch: 2.6.0
+- Datasets: 3.4.1
+- Tokenizers: 0.21.1
+
+## Citations
+
+
+
+Cite TRL as:
+    
+```bibtex
+@misc{vonwerra2022trl,
+	title        = {{TRL: Transformer Reinforcement Learning}},
+	author       = {Leandro von Werra and Younes Belkada and Lewis Tunstall and Edward Beeching and Tristan Thrush and Nathan Lambert and Shengyi Huang and Kashif Rasul and Quentin Gallouédec},
+	year         = 2020,
+	journal      = {GitHub repository},
+	publisher    = {GitHub},
+	howpublished = {\url{https://github.com/huggingface/trl}}
+}
+```

adapter_config.json

@@ -0,0 +1,37 @@
+{
+  "alpha_pattern": {},
+  "auto_mapping": null,
+  "base_model_name_or_path": "google/paligemma2-3b-pt-224",
+  "bias": "none",
+  "eva_config": null,
+  "exclude_modules": null,
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 16,
+  "lora_bias": false,
+  "lora_dropout": 0.05,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "r": 16,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "k_proj",
+    "down_proj",
+    "v_proj",
+    "q_proj",
+    "gate_proj",
+    "o_proj",
+    "up_proj"
+  ],
+  "task_type": "CAUSAL_LM",
+  "use_dora": false,
+  "use_rslora": false
+}

adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6be4b4f4bb2df72e2cb7de3a913546aecb8c7e69fbe1a6b9586df92895b6b352
+size 95091000

all_results.json

@@ -0,0 +1,13 @@
+{
+    "eval_loss": 1.53940749168396,
+    "eval_runtime": 1828.6515,
+    "eval_samples": 495720,
+    "eval_samples_per_second": 271.085,
+    "eval_steps_per_second": 2.118,
+    "total_flos": 3.9136612672224625e+18,
+    "train_loss": 1.9499194212634154,
+    "train_runtime": 7999.3345,
+    "train_samples": 798264,
+    "train_samples_per_second": 99.791,
+    "train_steps_per_second": 0.78
+}

checkpoint-3119/README.md

@@ -0,0 +1,202 @@
+---
+base_model: google/paligemma2-3b-pt-224
+library_name: peft
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.14.0

checkpoint-3119/adapter_config.json

@@ -0,0 +1,37 @@
+{
+  "alpha_pattern": {},
+  "auto_mapping": null,
+  "base_model_name_or_path": "google/paligemma2-3b-pt-224",
+  "bias": "none",
+  "eva_config": null,
+  "exclude_modules": null,
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 16,
+  "lora_bias": false,
+  "lora_dropout": 0.05,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "r": 16,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "k_proj",
+    "down_proj",
+    "v_proj",
+    "q_proj",
+    "gate_proj",
+    "o_proj",
+    "up_proj"
+  ],
+  "task_type": "CAUSAL_LM",
+  "use_dora": false,
+  "use_rslora": false
+}

checkpoint-3119/adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6be4b4f4bb2df72e2cb7de3a913546aecb8c7e69fbe1a6b9586df92895b6b352
+size 95091000

checkpoint-3119/optimizer.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8575e5a31ffda76bcfdfd4869ad1c154c8ed455d9092dad407844021d8b58522
+size 190464380

checkpoint-3119/preprocessor_config.json

@@ -0,0 +1,25 @@
+{
+  "do_convert_rgb": null,
+  "do_normalize": true,
+  "do_rescale": true,
+  "do_resize": true,
+  "image_mean": [
+    0.5,
+    0.5,
+    0.5
+  ],
+  "image_processor_type": "SiglipImageProcessor",
+  "image_seq_length": 256,
+  "image_std": [
+    0.5,
+    0.5,
+    0.5
+  ],
+  "processor_class": "PaliGemmaProcessor",
+  "resample": 3,
+  "rescale_factor": 0.00392156862745098,
+  "size": {
+    "height": 224,
+    "width": 224
+  }
+}

checkpoint-3119/rng_state_0.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:04af563841890f9c8b560cf6db1986a19c76e19d4aa67ea7b349d6eaeb941663
+size 15984

checkpoint-3119/rng_state_1.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a9a18a7e2476b38417256b04bc4d1430c4b333b1f8f45ae01146912b7244cd35
+size 15984

checkpoint-3119/rng_state_2.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5cf72f0ae9cfa868c0d359891f394058b2e294adbb476b9bce12d5c295c09b5b
+size 15984

checkpoint-3119/rng_state_3.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f28c4ad7eebc24f3afc465f4542fcd44656f7d4fd48da0bc9538735150633ae7
+size 15984

checkpoint-3119/rng_state_4.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:799a4314e0de8b7959d399ae82d6db686b4ae0913034b3900511c3781e04a785
+size 15984

checkpoint-3119/rng_state_5.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c369ab6e76142ede738ce07ab055e8ea0fa91fcffb397c253cdfa0a5a866510e
+size 15984

checkpoint-3119/rng_state_6.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c370545bb145612fe2fff7b80314a136d0749ed0c77ac2343e5a740030d9e7c2
+size 15984

checkpoint-3119/rng_state_7.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b2da3a2e4c00cb47e22bba9757bfe6891770da222ab9b1f7db87e6bfd157ed38
+size 15984

checkpoint-3119/scheduler.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ecc4c515c7b740e24b81f75d944f6e80b6541c0a472de59fdccbcba191a191ac
+size 1064

checkpoint-3119/special_tokens_map.json

@@ -0,0 +1,39 @@
+{
+  "additional_special_tokens": [
+    {
+      "content": "<image>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false
+    }
+  ],
+  "bos_token": {
+    "content": "<bos>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "<eos>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "<pad>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "unk_token": {
+    "content": "<unk>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

checkpoint-3119/tokenizer.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b648d11e0879b11659e6b4051f691752c0cef597a865c6fde5b318b9f68c1d05
+size 34600974

checkpoint-3119/trainer_state.json

@@ -0,0 +1,1290 @@
+{
+  "best_metric": 1.53940749168396,
+  "best_model_checkpoint": "data/paligemma2-3b-pt-224-sft-lora-vamos_10pct_gpt5_mini_cocoqa_localized_narratives_fixed/checkpoint-3119",
+  "epoch": 0.5000801667468334,
+  "eval_steps": 3119,
+  "global_step": 3119,
+  "is_hyper_param_search": false,
+  "is_local_process_zero": true,
+  "is_world_process_zero": true,
+  "log_history": [
+    {
+      "epoch": 0.000160333493666827,
+      "grad_norm": 5.209284782409668,
+      "learning_rate": 1.6025641025641025e-07,
+      "loss": 8.2025,
+      "mean_token_accuracy": 0.257368803024292,
+      "step": 1
+    },
+    {
+      "epoch": 0.00320666987333654,
+      "grad_norm": 5.846069812774658,
+      "learning_rate": 3.205128205128205e-06,
+      "loss": 8.8548,
+      "mean_token_accuracy": 0.2286823886005502,
+      "step": 20
+    },
+    {
+      "epoch": 0.00641333974667308,
+      "grad_norm": 6.527371406555176,
+      "learning_rate": 6.41025641025641e-06,
+      "loss": 8.6095,
+      "mean_token_accuracy": 0.236174263805151,
+      "step": 40
+    },
+    {
+      "epoch": 0.00962000962000962,
+      "grad_norm": 6.7473320960998535,
+      "learning_rate": 9.615384615384616e-06,
+      "loss": 7.0858,
+      "mean_token_accuracy": 0.24544834867119789,
+      "step": 60
+    },
+    {
+      "epoch": 0.01282667949334616,
+      "grad_norm": 3.6053383350372314,
+      "learning_rate": 1.282051282051282e-05,
+      "loss": 5.0951,
+      "mean_token_accuracy": 0.2442556582391262,
+      "step": 80
+    },
+    {
+      "epoch": 0.0160333493666827,
+      "grad_norm": 1.8141255378723145,
+      "learning_rate": 1.602564102564103e-05,
+      "loss": 4.0162,
+      "mean_token_accuracy": 0.29175431579351424,
+      "step": 100
+    },
+    {
+      "epoch": 0.01924001924001924,
+      "grad_norm": 1.9441587924957275,
+      "learning_rate": 1.923076923076923e-05,
+      "loss": 3.3901,
+      "mean_token_accuracy": 0.33477348536252977,
+      "step": 120
+    },
+    {
+      "epoch": 0.02244668911335578,
+      "grad_norm": 2.866448402404785,
+      "learning_rate": 2.2435897435897437e-05,
+      "loss": 3.0645,
+      "mean_token_accuracy": 0.36781402975320815,
+      "step": 140
+    },
+    {
+      "epoch": 0.02565335898669232,
+      "grad_norm": 6.173582553863525,
+      "learning_rate": 2.564102564102564e-05,
+      "loss": 2.8851,
+      "mean_token_accuracy": 0.3857575759291649,
+      "step": 160
+    },
+    {
+      "epoch": 0.02886002886002886,
+      "grad_norm": 2.632821798324585,
+      "learning_rate": 2.8846153846153845e-05,
+      "loss": 2.625,
+      "mean_token_accuracy": 0.42856554985046386,
+      "step": 180
+    },
+    {
+      "epoch": 0.0320666987333654,
+      "grad_norm": 3.801175594329834,
+      "learning_rate": 3.205128205128206e-05,
+      "loss": 2.5486,
+      "mean_token_accuracy": 0.44293054938316345,
+      "step": 200
+    },
+    {
+      "epoch": 0.03527336860670194,
+      "grad_norm": 2.8042311668395996,
+      "learning_rate": 3.525641025641026e-05,
+      "loss": 2.4267,
+      "mean_token_accuracy": 0.46162763684988023,
+      "step": 220
+    },
+    {
+      "epoch": 0.03848003848003848,
+      "grad_norm": 5.13622522354126,
+      "learning_rate": 3.846153846153846e-05,
+      "loss": 2.3457,
+      "mean_token_accuracy": 0.46204862743616104,
+      "step": 240
+    },
+    {
+      "epoch": 0.04168670835337502,
+      "grad_norm": 8.156232833862305,
+      "learning_rate": 4.166666666666667e-05,
+      "loss": 2.3276,
+      "mean_token_accuracy": 0.4628546267747879,
+      "step": 260
+    },
+    {
+      "epoch": 0.04489337822671156,
+      "grad_norm": 5.940677642822266,
+      "learning_rate": 4.4871794871794874e-05,
+      "loss": 2.3121,
+      "mean_token_accuracy": 0.4677517533302307,
+      "step": 280
+    },
+    {
+      "epoch": 0.0481000481000481,
+      "grad_norm": 3.2402215003967285,
+      "learning_rate": 4.8076923076923084e-05,
+      "loss": 2.2948,
+      "mean_token_accuracy": 0.4685308575630188,
+      "step": 300
+    },
+    {
+      "epoch": 0.05130671797338464,
+      "grad_norm": 4.654311656951904,
+      "learning_rate": 5.128205128205128e-05,
+      "loss": 2.2764,
+      "mean_token_accuracy": 0.46919394135475156,
+      "step": 320
+    },
+    {
+      "epoch": 0.05451338784672118,
+      "grad_norm": 6.181347370147705,
+      "learning_rate": 5.448717948717948e-05,
+      "loss": 2.2516,
+      "mean_token_accuracy": 0.4718445986509323,
+      "step": 340
+    },
+    {
+      "epoch": 0.05772005772005772,
+      "grad_norm": 4.977773189544678,
+      "learning_rate": 5.769230769230769e-05,
+      "loss": 2.3078,
+      "mean_token_accuracy": 0.46173652708530427,
+      "step": 360
+    },
+    {
+      "epoch": 0.06092672759339426,
+      "grad_norm": 2.649177074432373,
+      "learning_rate": 6.089743589743589e-05,
+      "loss": 2.2364,
+      "mean_token_accuracy": 0.4723318055272102,
+      "step": 380
+    },
+    {
+      "epoch": 0.0641333974667308,
+      "grad_norm": 2.475168228149414,
+      "learning_rate": 6.410256410256412e-05,
+      "loss": 2.2404,
+      "mean_token_accuracy": 0.4773345053195953,
+      "step": 400
+    },
+    {
+      "epoch": 0.06734006734006734,
+      "grad_norm": 3.376347541809082,
+      "learning_rate": 6.730769230769232e-05,
+      "loss": 2.1923,
+      "mean_token_accuracy": 0.48165750354528425,
+      "step": 420
+    },
+    {
+      "epoch": 0.07054673721340388,
+      "grad_norm": 2.1181466579437256,
+      "learning_rate": 7.051282051282052e-05,
+      "loss": 2.1944,
+      "mean_token_accuracy": 0.47836676687002183,
+      "step": 440
+    },
+    {
+      "epoch": 0.07375340708674041,
+      "grad_norm": 3.304994583129883,
+      "learning_rate": 7.371794871794872e-05,
+      "loss": 2.2055,
+      "mean_token_accuracy": 0.47685392051935194,
+      "step": 460
+    },
+    {
+      "epoch": 0.07696007696007696,
+      "grad_norm": 3.5151872634887695,
+      "learning_rate": 7.692307692307693e-05,
+      "loss": 2.171,
+      "mean_token_accuracy": 0.4798412054777145,
+      "step": 480
+    },
+    {
+      "epoch": 0.0801667468334135,
+      "grad_norm": 2.439455509185791,
+      "learning_rate": 8.012820512820514e-05,
+      "loss": 2.1592,
+      "mean_token_accuracy": 0.48496305793523786,
+      "step": 500
+    },
+    {
+      "epoch": 0.08337341670675004,
+      "grad_norm": 3.069302797317505,
+      "learning_rate": 8.333333333333334e-05,
+      "loss": 2.0884,
+      "mean_token_accuracy": 0.4890993982553482,
+      "step": 520
+    },
+    {
+      "epoch": 0.08658008658008658,
+      "grad_norm": 5.592751979827881,
+      "learning_rate": 8.653846153846155e-05,
+      "loss": 2.1425,
+      "mean_token_accuracy": 0.48478382378816604,
+      "step": 540
+    },
+    {
+      "epoch": 0.08978675645342311,
+      "grad_norm": 2.7817585468292236,
+      "learning_rate": 8.974358974358975e-05,
+      "loss": 2.1501,
+      "mean_token_accuracy": 0.4846224099397659,
+      "step": 560
+    },
+    {
+      "epoch": 0.09299342632675967,
+      "grad_norm": 3.0570271015167236,
+      "learning_rate": 9.294871794871795e-05,
+      "loss": 2.1297,
+      "mean_token_accuracy": 0.4879722028970718,
+      "step": 580
+    },
+    {
+      "epoch": 0.0962000962000962,
+      "grad_norm": 2.257469415664673,
+      "learning_rate": 9.615384615384617e-05,
+      "loss": 2.1088,
+      "mean_token_accuracy": 0.4889796242117882,
+      "step": 600
+    },
+    {
+      "epoch": 0.09940676607343274,
+      "grad_norm": 5.005424499511719,
+      "learning_rate": 9.935897435897437e-05,
+      "loss": 2.1248,
+      "mean_token_accuracy": 0.48695541769266126,
+      "step": 620
+    },
+    {
+      "epoch": 0.10261343594676928,
+      "grad_norm": 2.763110637664795,
+      "learning_rate": 9.999799512762136e-05,
+      "loss": 2.0852,
+      "mean_token_accuracy": 0.4945754811167717,
+      "step": 640
+    },
+    {
+      "epoch": 0.10582010582010581,
+      "grad_norm": 2.5508882999420166,
+      "learning_rate": 9.998985060913877e-05,
+      "loss": 2.1155,
+      "mean_token_accuracy": 0.48479859828948973,
+      "step": 660
+    },
+    {
+      "epoch": 0.10902677569344237,
+      "grad_norm": 1.9309931993484497,
+      "learning_rate": 9.99754421597847e-05,
+      "loss": 2.0774,
+      "mean_token_accuracy": 0.4935262441635132,
+      "step": 680
+    },
+    {
+      "epoch": 0.1122334455667789,
+      "grad_norm": 2.175483465194702,
+      "learning_rate": 9.995477158499624e-05,
+      "loss": 2.0698,
+      "mean_token_accuracy": 0.4930373579263687,
+      "step": 700
+    },
+    {
+      "epoch": 0.11544011544011544,
+      "grad_norm": 2.2829439640045166,
+      "learning_rate": 9.992784147488018e-05,
+      "loss": 2.064,
+      "mean_token_accuracy": 0.4933862343430519,
+      "step": 720
+    },
+    {
+      "epoch": 0.11864678531345198,
+      "grad_norm": 1.5809357166290283,
+      "learning_rate": 9.989465520388845e-05,
+      "loss": 2.0323,
+      "mean_token_accuracy": 0.5016711667180062,
+      "step": 740
+    },
+    {
+      "epoch": 0.12185345518678851,
+      "grad_norm": 2.230652332305908,
+      "learning_rate": 9.985521693039529e-05,
+      "loss": 2.075,
+      "mean_token_accuracy": 0.4931314527988434,
+      "step": 760
+    },
+    {
+      "epoch": 0.12506012506012507,
+      "grad_norm": 1.8354729413986206,
+      "learning_rate": 9.980953159617621e-05,
+      "loss": 2.0748,
+      "mean_token_accuracy": 0.4964547500014305,
+      "step": 780
+    },
+    {
+      "epoch": 0.1282667949334616,
+      "grad_norm": 2.15897274017334,
+      "learning_rate": 9.975760492578874e-05,
+      "loss": 2.0301,
+      "mean_token_accuracy": 0.4994760677218437,
+      "step": 800
+    },
+    {
+      "epoch": 0.13147346480679814,
+      "grad_norm": 2.6074705123901367,
+      "learning_rate": 9.969944342585508e-05,
+      "loss": 1.9838,
+      "mean_token_accuracy": 0.5075768753886223,
+      "step": 820
+    },
+    {
+      "epoch": 0.13468013468013468,
+      "grad_norm": 1.7504245042800903,
+      "learning_rate": 9.963505438424693e-05,
+      "loss": 2.0732,
+      "mean_token_accuracy": 0.49125437140464784,
+      "step": 840
+    },
+    {
+      "epoch": 0.13788680455347122,
+      "grad_norm": 2.214726686477661,
+      "learning_rate": 9.956444586917213e-05,
+      "loss": 2.0329,
+      "mean_token_accuracy": 0.4968407765030861,
+      "step": 860
+    },
+    {
+      "epoch": 0.14109347442680775,
+      "grad_norm": 1.6134241819381714,
+      "learning_rate": 9.948762672816377e-05,
+      "loss": 2.0392,
+      "mean_token_accuracy": 0.4989665150642395,
+      "step": 880
+    },
+    {
+      "epoch": 0.1443001443001443,
+      "grad_norm": 1.3124563694000244,
+      "learning_rate": 9.94046065869715e-05,
+      "loss": 2.0265,
+      "mean_token_accuracy": 0.5022040411829949,
+      "step": 900
+    },
+    {
+      "epoch": 0.14750681417348083,
+      "grad_norm": 2.182892084121704,
+      "learning_rate": 9.931539584835553e-05,
+      "loss": 2.0258,
+      "mean_token_accuracy": 0.4994161993265152,
+      "step": 920
+    },
+    {
+      "epoch": 0.1507134840468174,
+      "grad_norm": 2.158780813217163,
+      "learning_rate": 9.922000569078288e-05,
+      "loss": 2.038,
+      "mean_token_accuracy": 0.49309421628713607,
+      "step": 940
+    },
+    {
+      "epoch": 0.15392015392015393,
+      "grad_norm": 1.8986632823944092,
+      "learning_rate": 9.911844806702692e-05,
+      "loss": 2.0521,
+      "mean_token_accuracy": 0.4916775792837143,
+      "step": 960
+    },
+    {
+      "epoch": 0.15712682379349047,
+      "grad_norm": 2.2148079872131348,
+      "learning_rate": 9.901073570266946e-05,
+      "loss": 2.046,
+      "mean_token_accuracy": 0.4920832559466362,
+      "step": 980
+    },
+    {
+      "epoch": 0.160333493666827,
+      "grad_norm": 2.105100393295288,
+      "learning_rate": 9.889688209450627e-05,
+      "loss": 2.0154,
+      "mean_token_accuracy": 0.5017145529389382,
+      "step": 1000
+    },
+    {
+      "epoch": 0.16354016354016354,
+      "grad_norm": 1.9177722930908203,
+      "learning_rate": 9.877690150885588e-05,
+      "loss": 2.0133,
+      "mean_token_accuracy": 0.502875691652298,
+      "step": 1020
+    },
+    {
+      "epoch": 0.16674683341350008,
+      "grad_norm": 1.3427209854125977,
+      "learning_rate": 9.865080897977188e-05,
+      "loss": 1.9948,
+      "mean_token_accuracy": 0.505776722729206,
+      "step": 1040
+    },
+    {
+      "epoch": 0.16995350328683662,
+      "grad_norm": 2.0759928226470947,
+      "learning_rate": 9.851862030715919e-05,
+      "loss": 1.9401,
+      "mean_token_accuracy": 0.5120617598295212,
+      "step": 1060
+    },
+    {
+      "epoch": 0.17316017316017315,
+      "grad_norm": 1.2468688488006592,
+      "learning_rate": 9.838035205479419e-05,
+      "loss": 1.9913,
+      "mean_token_accuracy": 0.5054045274853707,
+      "step": 1080
+    },
+    {
+      "epoch": 0.1763668430335097,
+      "grad_norm": 1.2889540195465088,
+      "learning_rate": 9.823602154824927e-05,
+      "loss": 1.9963,
+      "mean_token_accuracy": 0.505834324657917,
+      "step": 1100
+    },
+    {
+      "epoch": 0.17957351290684623,
+      "grad_norm": 1.4190268516540527,
+      "learning_rate": 9.808564687272178e-05,
+      "loss": 1.9797,
+      "mean_token_accuracy": 0.5044485762715339,
+      "step": 1120
+    },
+    {
+      "epoch": 0.1827801827801828,
+      "grad_norm": 1.7106951475143433,
+      "learning_rate": 9.792924687076801e-05,
+      "loss": 1.9863,
+      "mean_token_accuracy": 0.5070938616991043,
+      "step": 1140
+    },
+    {
+      "epoch": 0.18598685265351933,
+      "grad_norm": 1.247165322303772,
+      "learning_rate": 9.776684113994204e-05,
+      "loss": 1.9723,
+      "mean_token_accuracy": 0.5093738153576851,
+      "step": 1160
+    },
+    {
+      "epoch": 0.18919352252685587,
+      "grad_norm": 1.4030295610427856,
+      "learning_rate": 9.759845003034007e-05,
+      "loss": 1.9756,
+      "mean_token_accuracy": 0.5056015223264694,
+      "step": 1180
+    },
+    {
+      "epoch": 0.1924001924001924,
+      "grad_norm": 1.243714690208435,
+      "learning_rate": 9.74240946420506e-05,
+      "loss": 1.9513,
+      "mean_token_accuracy": 0.5091609105467796,
+      "step": 1200
+    },
+    {
+      "epoch": 0.19560686227352894,
+      "grad_norm": 0.9765139222145081,
+      "learning_rate": 9.724379682251036e-05,
+      "loss": 1.9808,
+      "mean_token_accuracy": 0.50465277582407,
+      "step": 1220
+    },
+    {
+      "epoch": 0.19881353214686548,
+      "grad_norm": 1.4885112047195435,
+      "learning_rate": 9.705757916376682e-05,
+      "loss": 1.9993,
+      "mean_token_accuracy": 0.5045629054307937,
+      "step": 1240
+    },
+    {
+      "epoch": 0.20202020202020202,
+      "grad_norm": 1.258948802947998,
+      "learning_rate": 9.686546499964731e-05,
+      "loss": 1.9868,
+      "mean_token_accuracy": 0.5066259905695916,
+      "step": 1260
+    },
+    {
+      "epoch": 0.20522687189353855,
+      "grad_norm": 1.192187786102295,
+      "learning_rate": 9.666747840283512e-05,
+      "loss": 1.9444,
+      "mean_token_accuracy": 0.5078241482377053,
+      "step": 1280
+    },
+    {
+      "epoch": 0.2084335417668751,
+      "grad_norm": 1.1449286937713623,
+      "learning_rate": 9.646364418185325e-05,
+      "loss": 1.9964,
+      "mean_token_accuracy": 0.5036799103021622,
+      "step": 1300
+    },
+    {
+      "epoch": 0.21164021164021163,
+      "grad_norm": 1.2843917608261108,
+      "learning_rate": 9.62539878779556e-05,
+      "loss": 1.9857,
+      "mean_token_accuracy": 0.5052528932690621,
+      "step": 1320
+    },
+    {
+      "epoch": 0.2148468815135482,
+      "grad_norm": 1.1665757894515991,
+      "learning_rate": 9.603853576192674e-05,
+      "loss": 1.9676,
+      "mean_token_accuracy": 0.5088991582393646,
+      "step": 1340
+    },
+    {
+      "epoch": 0.21805355138688473,
+      "grad_norm": 1.0627950429916382,
+      "learning_rate": 9.581731483078995e-05,
+      "loss": 1.9494,
+      "mean_token_accuracy": 0.512041249871254,
+      "step": 1360
+    },
+    {
+      "epoch": 0.22126022126022127,
+      "grad_norm": 1.3954745531082153,
+      "learning_rate": 9.559035280442443e-05,
+      "loss": 1.9352,
+      "mean_token_accuracy": 0.5163529127836227,
+      "step": 1380
+    },
+    {
+      "epoch": 0.2244668911335578,
+      "grad_norm": 2.581529378890991,
+      "learning_rate": 9.535767812209185e-05,
+      "loss": 1.966,
+      "mean_token_accuracy": 0.5078106135129928,
+      "step": 1400
+    },
+    {
+      "epoch": 0.22767356100689434,
+      "grad_norm": 1.995743751525879,
+      "learning_rate": 9.511931993887289e-05,
+      "loss": 1.9278,
+      "mean_token_accuracy": 0.5141927003860474,
+      "step": 1420
+    },
+    {
+      "epoch": 0.23088023088023088,
+      "grad_norm": 1.3608646392822266,
+      "learning_rate": 9.487530812201384e-05,
+      "loss": 1.9464,
+      "mean_token_accuracy": 0.5119614064693451,
+      "step": 1440
+    },
+    {
+      "epoch": 0.23408690075356742,
+      "grad_norm": 2.1510233879089355,
+      "learning_rate": 9.462567324718428e-05,
+      "loss": 1.972,
+      "mean_token_accuracy": 0.5053435668349266,
+      "step": 1460
+    },
+    {
+      "epoch": 0.23729357062690395,
+      "grad_norm": 1.3489898443222046,
+      "learning_rate": 9.437044659464568e-05,
+      "loss": 1.9406,
+      "mean_token_accuracy": 0.5107260003685952,
+      "step": 1480
+    },
+    {
+      "epoch": 0.2405002405002405,
+      "grad_norm": 2.1510982513427734,
+      "learning_rate": 9.410966014533196e-05,
+      "loss": 1.9501,
+      "mean_token_accuracy": 0.5109146684408188,
+      "step": 1500
+    },
+    {
+      "epoch": 0.24370691037357703,
+      "grad_norm": 1.3027640581130981,
+      "learning_rate": 9.384334657684207e-05,
+      "loss": 1.9162,
+      "mean_token_accuracy": 0.5180920317769051,
+      "step": 1520
+    },
+    {
+      "epoch": 0.24691358024691357,
+      "grad_norm": 1.6839220523834229,
+      "learning_rate": 9.357153925934543e-05,
+      "loss": 1.932,
+      "mean_token_accuracy": 0.5122836321592331,
+      "step": 1540
+    },
+    {
+      "epoch": 0.25012025012025013,
+      "grad_norm": 1.5489161014556885,
+      "learning_rate": 9.329427225140042e-05,
+      "loss": 1.9287,
+      "mean_token_accuracy": 0.5156795844435692,
+      "step": 1560
+    },
+    {
+      "epoch": 0.25332691999358664,
+      "grad_norm": 1.1957494020462036,
+      "learning_rate": 9.301158029568684e-05,
+      "loss": 1.8905,
+      "mean_token_accuracy": 0.5207541212439537,
+      "step": 1580
+    },
+    {
+      "epoch": 0.2565335898669232,
+      "grad_norm": 1.2413902282714844,
+      "learning_rate": 9.272349881465236e-05,
+      "loss": 1.9309,
+      "mean_token_accuracy": 0.5141352891921998,
+      "step": 1600
+    },
+    {
+      "epoch": 0.2597402597402597,
+      "grad_norm": 1.3737406730651855,
+      "learning_rate": 9.243006390607403e-05,
+      "loss": 1.9565,
+      "mean_token_accuracy": 0.5097300469875335,
+      "step": 1620
+    },
+    {
+      "epoch": 0.2629469296135963,
+      "grad_norm": 1.24127197265625,
+      "learning_rate": 9.213131233853511e-05,
+      "loss": 1.945,
+      "mean_token_accuracy": 0.513680812716484,
+      "step": 1640
+    },
+    {
+      "epoch": 0.26615359948693285,
+      "grad_norm": 1.2227799892425537,
+      "learning_rate": 9.182728154681772e-05,
+      "loss": 1.945,
+      "mean_token_accuracy": 0.5159043714404106,
+      "step": 1660
+    },
+    {
+      "epoch": 0.26936026936026936,
+      "grad_norm": 1.4925428628921509,
+      "learning_rate": 9.151800962721218e-05,
+      "loss": 1.959,
+      "mean_token_accuracy": 0.5081851810216904,
+      "step": 1680
+    },
+    {
+      "epoch": 0.2725669392336059,
+      "grad_norm": 1.154524564743042,
+      "learning_rate": 9.12035353327434e-05,
+      "loss": 1.9162,
+      "mean_token_accuracy": 0.517485597729683,
+      "step": 1700
+    },
+    {
+      "epoch": 0.27577360910694243,
+      "grad_norm": 1.1241073608398438,
+      "learning_rate": 9.08838980683149e-05,
+      "loss": 1.9477,
+      "mean_token_accuracy": 0.5126137733459473,
+      "step": 1720
+    },
+    {
+      "epoch": 0.278980278980279,
+      "grad_norm": 1.8384186029434204,
+      "learning_rate": 9.055913788577128e-05,
+      "loss": 1.9197,
+      "mean_token_accuracy": 0.5140128180384635,
+      "step": 1740
+    },
+    {
+      "epoch": 0.2821869488536155,
+      "grad_norm": 1.3156044483184814,
+      "learning_rate": 9.02292954788796e-05,
+      "loss": 1.9079,
+      "mean_token_accuracy": 0.5170748963952064,
+      "step": 1760
+    },
+    {
+      "epoch": 0.28539361872695207,
+      "grad_norm": 1.449820876121521,
+      "learning_rate": 8.98944121782301e-05,
+      "loss": 1.9342,
+      "mean_token_accuracy": 0.5114595577120781,
+      "step": 1780
+    },
+    {
+      "epoch": 0.2886002886002886,
+      "grad_norm": 1.6988258361816406,
+      "learning_rate": 8.955452994605753e-05,
+      "loss": 1.9449,
+      "mean_token_accuracy": 0.5123026236891747,
+      "step": 1800
+    },
+    {
+      "epoch": 0.29180695847362514,
+      "grad_norm": 1.0350065231323242,
+      "learning_rate": 8.920969137098303e-05,
+      "loss": 1.9213,
+      "mean_token_accuracy": 0.5155784547328949,
+      "step": 1820
+    },
+    {
+      "epoch": 0.29501362834696165,
+      "grad_norm": 1.1465715169906616,
+      "learning_rate": 8.885993966267753e-05,
+      "loss": 1.8896,
+      "mean_token_accuracy": 0.5216770812869072,
+      "step": 1840
+    },
+    {
+      "epoch": 0.2982202982202982,
+      "grad_norm": 1.1059490442276,
+      "learning_rate": 8.850531864644749e-05,
+      "loss": 1.8978,
+      "mean_token_accuracy": 0.5205768331885338,
+      "step": 1860
+    },
+    {
+      "epoch": 0.3014269680936348,
+      "grad_norm": 1.199690818786621,
+      "learning_rate": 8.81458727577434e-05,
+      "loss": 1.9053,
+      "mean_token_accuracy": 0.5151138812303543,
+      "step": 1880
+    },
+    {
+      "epoch": 0.3046336379669713,
+      "grad_norm": 1.373304009437561,
+      "learning_rate": 8.77816470365918e-05,
+      "loss": 1.9276,
+      "mean_token_accuracy": 0.5131517440080643,
+      "step": 1900
+    },
+    {
+      "epoch": 0.30784030784030786,
+      "grad_norm": 1.4349356889724731,
+      "learning_rate": 8.741268712195166e-05,
+      "loss": 1.8921,
+      "mean_token_accuracy": 0.5225584238767624,
+      "step": 1920
+    },
+    {
+      "epoch": 0.31104697771364437,
+      "grad_norm": 1.0764795541763306,
+      "learning_rate": 8.703903924599548e-05,
+      "loss": 1.8639,
+      "mean_token_accuracy": 0.5256009042263031,
+      "step": 1940
+    },
+    {
+      "epoch": 0.31425364758698093,
+      "grad_norm": 1.946021318435669,
+      "learning_rate": 8.666075022831643e-05,
+      "loss": 1.881,
+      "mean_token_accuracy": 0.5257818832993507,
+      "step": 1960
+    },
+    {
+      "epoch": 0.31746031746031744,
+      "grad_norm": 1.2787318229675293,
+      "learning_rate": 8.627786747006145e-05,
+      "loss": 1.8966,
+      "mean_token_accuracy": 0.523352475464344,
+      "step": 1980
+    },
+    {
+      "epoch": 0.320666987333654,
+      "grad_norm": 1.7891284227371216,
+      "learning_rate": 8.589043894799185e-05,
+      "loss": 1.93,
+      "mean_token_accuracy": 0.5103022322058678,
+      "step": 2000
+    },
+    {
+      "epoch": 0.3238736572069905,
+      "grad_norm": 1.8391680717468262,
+      "learning_rate": 8.549851320847156e-05,
+      "loss": 1.8785,
+      "mean_token_accuracy": 0.5231836125254631,
+      "step": 2020
+    },
+    {
+      "epoch": 0.3270803270803271,
+      "grad_norm": 1.2025116682052612,
+      "learning_rate": 8.510213936138403e-05,
+      "loss": 1.8816,
+      "mean_token_accuracy": 0.5240169554948807,
+      "step": 2040
+    },
+    {
+      "epoch": 0.33028699695366365,
+      "grad_norm": 1.1192851066589355,
+      "learning_rate": 8.470136707397868e-05,
+      "loss": 1.8948,
+      "mean_token_accuracy": 0.5194600358605385,
+      "step": 2060
+    },
+    {
+      "epoch": 0.33349366682700016,
+      "grad_norm": 1.521708369255066,
+      "learning_rate": 8.42962465646473e-05,
+      "loss": 1.8973,
+      "mean_token_accuracy": 0.5208257511258125,
+      "step": 2080
+    },
+    {
+      "epoch": 0.3367003367003367,
+      "grad_norm": 1.6692718267440796,
+      "learning_rate": 8.388682859663152e-05,
+      "loss": 1.887,
+      "mean_token_accuracy": 0.5233332857489585,
+      "step": 2100
+    },
+    {
+      "epoch": 0.33990700657367323,
+      "grad_norm": 1.4780927896499634,
+      "learning_rate": 8.347316447166195e-05,
+      "loss": 1.8915,
+      "mean_token_accuracy": 0.523545055091381,
+      "step": 2120
+    },
+    {
+      "epoch": 0.3431136764470098,
+      "grad_norm": 1.7447547912597656,
+      "learning_rate": 8.305530602352985e-05,
+      "loss": 1.8826,
+      "mean_token_accuracy": 0.5198934078216553,
+      "step": 2140
+    },
+    {
+      "epoch": 0.3463203463203463,
+      "grad_norm": 1.1051859855651855,
+      "learning_rate": 8.263330561159222e-05,
+      "loss": 1.8792,
+      "mean_token_accuracy": 0.5219237297773361,
+      "step": 2160
+    },
+    {
+      "epoch": 0.34952701619368287,
+      "grad_norm": 1.2067945003509521,
+      "learning_rate": 8.220721611421087e-05,
+      "loss": 1.8989,
+      "mean_token_accuracy": 0.5167458400130271,
+      "step": 2180
+    },
+    {
+      "epoch": 0.3527336860670194,
+      "grad_norm": 1.3066425323486328,
+      "learning_rate": 8.177709092212663e-05,
+      "loss": 1.8421,
+      "mean_token_accuracy": 0.5322213441133499,
+      "step": 2200
+    },
+    {
+      "epoch": 0.35594035594035595,
+      "grad_norm": 1.0397696495056152,
+      "learning_rate": 8.134298393176916e-05,
+      "loss": 1.9187,
+      "mean_token_accuracy": 0.5156819596886635,
+      "step": 2220
+    },
+    {
+      "epoch": 0.35914702581369246,
+      "grad_norm": 1.3601312637329102,
+      "learning_rate": 8.090494953850365e-05,
+      "loss": 1.8785,
+      "mean_token_accuracy": 0.5235566765069961,
+      "step": 2240
+    },
+    {
+      "epoch": 0.362353695687029,
+      "grad_norm": 1.2964980602264404,
+      "learning_rate": 8.046304262981478e-05,
+      "loss": 1.8908,
+      "mean_token_accuracy": 0.5194398432970047,
+      "step": 2260
+    },
+    {
+      "epoch": 0.3655603655603656,
+      "grad_norm": 1.3470860719680786,
+      "learning_rate": 8.001731857842906e-05,
+      "loss": 1.9171,
+      "mean_token_accuracy": 0.51639633923769,
+      "step": 2280
+    },
+    {
+      "epoch": 0.3687670354337021,
+      "grad_norm": 1.3893637657165527,
+      "learning_rate": 7.956783323537652e-05,
+      "loss": 1.8682,
+      "mean_token_accuracy": 0.5249380469322205,
+      "step": 2300
+    },
+    {
+      "epoch": 0.37197370530703866,
+      "grad_norm": 1.5145325660705566,
+      "learning_rate": 7.911464292299222e-05,
+      "loss": 1.8457,
+      "mean_token_accuracy": 0.527319672703743,
+      "step": 2320
+    },
+    {
+      "epoch": 0.37518037518037517,
+      "grad_norm": 1.0269887447357178,
+      "learning_rate": 7.865780442785891e-05,
+      "loss": 1.8765,
+      "mean_token_accuracy": 0.5214496284723282,
+      "step": 2340
+    },
+    {
+      "epoch": 0.37838704505371173,
+      "grad_norm": 1.409474492073059,
+      "learning_rate": 7.81973749936914e-05,
+      "loss": 1.8571,
+      "mean_token_accuracy": 0.5259096622467041,
+      "step": 2360
+    },
+    {
+      "epoch": 0.38159371492704824,
+      "grad_norm": 1.0306053161621094,
+      "learning_rate": 7.77334123141638e-05,
+      "loss": 1.9028,
+      "mean_token_accuracy": 0.5194058448076249,
+      "step": 2380
+    },
+    {
+      "epoch": 0.3848003848003848,
+      "grad_norm": 1.076977014541626,
+      "learning_rate": 7.726597452568007e-05,
+      "loss": 1.8671,
+      "mean_token_accuracy": 0.5254542633891106,
+      "step": 2400
+    },
+    {
+      "epoch": 0.3880070546737213,
+      "grad_norm": 2.2276298999786377,
+      "learning_rate": 7.679512020008948e-05,
+      "loss": 1.8584,
+      "mean_token_accuracy": 0.5245337873697281,
+      "step": 2420
+    },
+    {
+      "epoch": 0.3912137245470579,
+      "grad_norm": 1.356619119644165,
+      "learning_rate": 7.632090833734723e-05,
+      "loss": 1.8601,
+      "mean_token_accuracy": 0.5262611448764801,
+      "step": 2440
+    },
+    {
+      "epoch": 0.3944203944203944,
+      "grad_norm": 1.5250309705734253,
+      "learning_rate": 7.584339835812151e-05,
+      "loss": 1.8671,
+      "mean_token_accuracy": 0.520966699719429,
+      "step": 2460
+    },
+    {
+      "epoch": 0.39762706429373096,
+      "grad_norm": 0.9932346940040588,
+      "learning_rate": 7.536265009634789e-05,
+      "loss": 1.8354,
+      "mean_token_accuracy": 0.5302907466888428,
+      "step": 2480
+    },
+    {
+      "epoch": 0.4008337341670675,
+      "grad_norm": 1.4524379968643188,
+      "learning_rate": 7.487872379173187e-05,
+      "loss": 1.8471,
+      "mean_token_accuracy": 0.527701361477375,
+      "step": 2500
+    },
+    {
+      "epoch": 0.40404040404040403,
+      "grad_norm": 1.6131216287612915,
+      "learning_rate": 7.439168008220057e-05,
+      "loss": 1.8791,
+      "mean_token_accuracy": 0.5199943497776985,
+      "step": 2520
+    },
+    {
+      "epoch": 0.4072470739137406,
+      "grad_norm": 1.2683628797531128,
+      "learning_rate": 7.390157999630463e-05,
+      "loss": 1.862,
+      "mean_token_accuracy": 0.5257236033678054,
+      "step": 2540
+    },
+    {
+      "epoch": 0.4104537437870771,
+      "grad_norm": 1.3786700963974,
+      "learning_rate": 7.340848494557098e-05,
+      "loss": 1.8329,
+      "mean_token_accuracy": 0.5322318792343139,
+      "step": 2560
+    },
+    {
+      "epoch": 0.4136604136604137,
+      "grad_norm": 1.2964134216308594,
+      "learning_rate": 7.291245671680781e-05,
+      "loss": 1.8378,
+      "mean_token_accuracy": 0.5295253932476044,
+      "step": 2580
+    },
+    {
+      "epoch": 0.4168670835337502,
+      "grad_norm": 1.2435472011566162,
+      "learning_rate": 7.241355746436243e-05,
+      "loss": 1.843,
+      "mean_token_accuracy": 0.528707692027092,
+      "step": 2600
+    },
+    {
+      "epoch": 0.42007375340708675,
+      "grad_norm": 1.6479510068893433,
+      "learning_rate": 7.191184970233299e-05,
+      "loss": 1.8559,
+      "mean_token_accuracy": 0.5243293821811676,
+      "step": 2620
+    },
+    {
+      "epoch": 0.42328042328042326,
+      "grad_norm": 1.4122897386550903,
+      "learning_rate": 7.140739629673531e-05,
+      "loss": 1.8584,
+      "mean_token_accuracy": 0.5285557866096496,
+      "step": 2640
+    },
+    {
+      "epoch": 0.4264870931537598,
+      "grad_norm": 1.2838547229766846,
+      "learning_rate": 7.09002604576255e-05,
+      "loss": 1.8609,
+      "mean_token_accuracy": 0.5250947296619415,
+      "step": 2660
+    },
+    {
+      "epoch": 0.4296937630270964,
+      "grad_norm": 1.5833513736724854,
+      "learning_rate": 7.03905057311794e-05,
+      "loss": 1.8246,
+      "mean_token_accuracy": 0.5320859342813492,
+      "step": 2680
+    },
+    {
+      "epoch": 0.4329004329004329,
+      "grad_norm": 1.5128042697906494,
+      "learning_rate": 6.987819599173007e-05,
+      "loss": 1.8332,
+      "mean_token_accuracy": 0.5332021147012711,
+      "step": 2700
+    },
+    {
+      "epoch": 0.43610710277376946,
+      "grad_norm": 1.0662071704864502,
+      "learning_rate": 6.936339543376411e-05,
+      "loss": 1.8757,
+      "mean_token_accuracy": 0.5246989518404007,
+      "step": 2720
+    },
+    {
+      "epoch": 0.43931377264710597,
+      "grad_norm": 1.0444306135177612,
+      "learning_rate": 6.884616856387768e-05,
+      "loss": 1.8411,
+      "mean_token_accuracy": 0.5272899508476258,
+      "step": 2740
+    },
+    {
+      "epoch": 0.44252044252044254,
+      "grad_norm": 1.4123362302780151,
+      "learning_rate": 6.832658019269374e-05,
+      "loss": 1.8165,
+      "mean_token_accuracy": 0.5287312120199203,
+      "step": 2760
+    },
+    {
+      "epoch": 0.44572711239377905,
+      "grad_norm": 1.2641295194625854,
+      "learning_rate": 6.780469542674088e-05,
+      "loss": 1.8581,
+      "mean_token_accuracy": 0.5240399688482285,
+      "step": 2780
+    },
+    {
+      "epoch": 0.4489337822671156,
+      "grad_norm": 1.055584192276001,
+      "learning_rate": 6.728057966029529e-05,
+      "loss": 1.7969,
+      "mean_token_accuracy": 0.536593297123909,
+      "step": 2800
+    },
+    {
+      "epoch": 0.4521404521404521,
+      "grad_norm": 1.2474592924118042,
+      "learning_rate": 6.675429856718652e-05,
+      "loss": 1.8295,
+      "mean_token_accuracy": 0.5313476771116257,
+      "step": 2820
+    },
+    {
+      "epoch": 0.4553471220137887,
+      "grad_norm": 1.1250567436218262,
+      "learning_rate": 6.62259180925684e-05,
+      "loss": 1.8102,
+      "mean_token_accuracy": 0.5341511443257332,
+      "step": 2840
+    },
+    {
+      "epoch": 0.4585537918871252,
+      "grad_norm": 1.2222751379013062,
+      "learning_rate": 6.569550444465568e-05,
+      "loss": 1.8343,
+      "mean_token_accuracy": 0.5306926101446152,
+      "step": 2860
+    },
+    {
+      "epoch": 0.46176046176046176,
+      "grad_norm": 1.24721097946167,
+      "learning_rate": 6.516312408642804e-05,
+      "loss": 1.8142,
+      "mean_token_accuracy": 0.5346602320671081,
+      "step": 2880
+    },
+    {
+      "epoch": 0.4649671316337983,
+      "grad_norm": 1.3725354671478271,
+      "learning_rate": 6.462884372730191e-05,
+      "loss": 1.8515,
+      "mean_token_accuracy": 0.525643666088581,
+      "step": 2900
+    },
+    {
+      "epoch": 0.46817380150713483,
+      "grad_norm": 1.5340793132781982,
+      "learning_rate": 6.409273031477152e-05,
+      "loss": 1.8405,
+      "mean_token_accuracy": 0.5296775609254837,
+      "step": 2920
+    },
+    {
+      "epoch": 0.4713804713804714,
+      "grad_norm": 1.4171345233917236,
+      "learning_rate": 6.355485102602012e-05,
+      "loss": 1.801,
+      "mean_token_accuracy": 0.5374062955379486,
+      "step": 2940
+    },
+    {
+      "epoch": 0.4745871412538079,
+      "grad_norm": 1.0726767778396606,
+      "learning_rate": 6.301527325950242e-05,
+      "loss": 1.8177,
+      "mean_token_accuracy": 0.5328266575932503,
+      "step": 2960
+    },
+    {
+      "epoch": 0.4777938111271445,
+      "grad_norm": 2.005058526992798,
+      "learning_rate": 6.247406462649927e-05,
+      "loss": 1.8109,
+      "mean_token_accuracy": 0.5314099788665771,
+      "step": 2980
+    },
+    {
+      "epoch": 0.481000481000481,
+      "grad_norm": 1.5543380975723267,
+      "learning_rate": 6.193129294264569e-05,
+      "loss": 1.8432,
+      "mean_token_accuracy": 0.5261458426713943,
+      "step": 3000
+    },
+    {
+      "epoch": 0.48420715087381755,
+      "grad_norm": 1.353722333908081,
+      "learning_rate": 6.138702621943334e-05,
+      "loss": 1.8178,
+      "mean_token_accuracy": 0.5276563599705696,
+      "step": 3020
+    },
+    {
+      "epoch": 0.48741382074715406,
+      "grad_norm": 1.1928757429122925,
+      "learning_rate": 6.0841332655688344e-05,
+      "loss": 1.8318,
+      "mean_token_accuracy": 0.5304355949163437,
+      "step": 3040
+    },
+    {
+      "epoch": 0.4906204906204906,
+      "grad_norm": 1.672419786453247,
+      "learning_rate": 6.029428062902571e-05,
+      "loss": 1.7971,
+      "mean_token_accuracy": 0.5380920082330704,
+      "step": 3060
+    },
+    {
+      "epoch": 0.49382716049382713,
+      "grad_norm": 1.231449842453003,
+      "learning_rate": 5.97459386872814e-05,
+      "loss": 1.8096,
+      "mean_token_accuracy": 0.5327497273683548,
+      "step": 3080
+    },
+    {
+      "epoch": 0.4970338303671637,
+      "grad_norm": 1.3776133060455322,
+      "learning_rate": 5.919637553992293e-05,
+      "loss": 1.8465,
+      "mean_token_accuracy": 0.5253860056400299,
+      "step": 3100
+    },
+    {
+      "epoch": 0.5000801667468334,
+      "eval_loss": 1.53940749168396,
+      "eval_mean_token_accuracy": 0.5933501680610359,
+      "eval_runtime": 1941.3293,
+      "eval_samples_per_second": 255.351,
+      "eval_steps_per_second": 1.995,
+      "step": 3119
+    }
+  ],
+  "logging_steps": 20,
+  "max_steps": 6237,
+  "num_input_tokens_seen": 0,
+  "num_train_epochs": 1,
+  "save_steps": 3119,
+  "stateful_callbacks": {
+    "TrainerControl": {
+      "args": {
+        "should_epoch_stop": false,
+        "should_evaluate": false,
+        "should_log": false,
+        "should_save": true,
+        "should_training_stop": false
+      },
+      "attributes": {}
+    }
+  },
+  "total_flos": 1.9570525453439468e+18,
+  "train_batch_size": 16,
+  "trial_name": null,
+  "trial_params": null
+}

checkpoint-3119/training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:aad656740548acb2cf8355c0bb58f2d3acfb74bed369d470a571070c08698a6c
+size 5880

checkpoint-6237/README.md

@@ -0,0 +1,202 @@
+---
+base_model: google/paligemma2-3b-pt-224
+library_name: peft
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.14.0

checkpoint-6237/adapter_config.json

@@ -0,0 +1,37 @@
+{
+  "alpha_pattern": {},
+  "auto_mapping": null,
+  "base_model_name_or_path": "google/paligemma2-3b-pt-224",
+  "bias": "none",
+  "eva_config": null,
+  "exclude_modules": null,
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 16,
+  "lora_bias": false,
+  "lora_dropout": 0.05,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "r": 16,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "k_proj",
+    "down_proj",
+    "v_proj",
+    "q_proj",
+    "gate_proj",
+    "o_proj",
+    "up_proj"
+  ],
+  "task_type": "CAUSAL_LM",
+  "use_dora": false,
+  "use_rslora": false
+}

checkpoint-6237/adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5d683bb24b341d958fb9d71c1e8472819d09d5f0a9f049789397bc604811fe36
+size 95091000

checkpoint-6237/optimizer.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:bd9a79b410efd0b42dc41ba6f672f4ffee035f0160b3f2695a9e549961f516a7
+size 190464380

checkpoint-6237/preprocessor_config.json

@@ -0,0 +1,25 @@
+{
+  "do_convert_rgb": null,
+  "do_normalize": true,
+  "do_rescale": true,
+  "do_resize": true,
+  "image_mean": [
+    0.5,
+    0.5,
+    0.5
+  ],
+  "image_processor_type": "SiglipImageProcessor",
+  "image_seq_length": 256,
+  "image_std": [
+    0.5,
+    0.5,
+    0.5
+  ],
+  "processor_class": "PaliGemmaProcessor",
+  "resample": 3,
+  "rescale_factor": 0.00392156862745098,
+  "size": {
+    "height": 224,
+    "width": 224
+  }
+}

checkpoint-6237/rng_state_0.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:fccb2fb54d9a5df26df7782da6d75867e9664903bb442e3acd1ae56d80feb397
+size 15984

checkpoint-6237/rng_state_1.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4695abe3ecabb35687e9fd27d60fe3db4fef0de9f4918f60d8ed610a9b93772f
+size 15984

checkpoint-6237/rng_state_2.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:62d2d9d220c601ba1734729cd560fc2a261addd3b6b2777e702c3773b520184b
+size 15984

checkpoint-6237/rng_state_3.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ea15159ed8b224ab2a3649b0f228d3b49cc94f8ed11142f1b58d6d4f87963161
+size 15984

checkpoint-6237/rng_state_4.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:75277f36cc267d12721c24a4e496596347be4fa07c39ed6a3990dfd8030f3c15
+size 15984

checkpoint-6237/rng_state_5.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a6f15e2e2881d26c3f17cf15b13cee969cd6a64c7028197061f71f797a3b88bd
+size 15984

checkpoint-6237/rng_state_6.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:da2bd8fbe2c292ae584499f8550029b9b946fc0be5beb7318660461f229750cb
+size 15984

checkpoint-6237/rng_state_7.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:574bb62dc75d4b29de879a656cdf9c548f7232ca5ed18b918c19157280114e65
+size 15984

checkpoint-6237/scheduler.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:946c531822c5128595d486c4af9725a1d9841e824b7ce7d6f457265b98a08947
+size 1064

checkpoint-6237/special_tokens_map.json

@@ -0,0 +1,39 @@
+{
+  "additional_special_tokens": [
+    {
+      "content": "<image>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false
+    }
+  ],
+  "bos_token": {
+    "content": "<bos>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "<eos>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "<pad>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "unk_token": {
+    "content": "<unk>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

checkpoint-6237/tokenizer.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b648d11e0879b11659e6b4051f691752c0cef597a865c6fde5b318b9f68c1d05
+size 34600974

checkpoint-6237/trainer_state.json

@@ -0,0 +1,2538 @@
+{
+  "best_metric": 1.53940749168396,
+  "best_model_checkpoint": "data/paligemma2-3b-pt-224-sft-lora-vamos_10pct_gpt5_mini_cocoqa_localized_narratives_fixed/checkpoint-3119",
+  "epoch": 1.0,
+  "eval_steps": 3119,
+  "global_step": 6237,
+  "is_hyper_param_search": false,
+  "is_local_process_zero": true,
+  "is_world_process_zero": true,
+  "log_history": [
+    {
+      "epoch": 0.000160333493666827,
+      "grad_norm": 5.209284782409668,
+      "learning_rate": 1.6025641025641025e-07,
+      "loss": 8.2025,
+      "mean_token_accuracy": 0.257368803024292,
+      "step": 1
+    },
+    {
+      "epoch": 0.00320666987333654,
+      "grad_norm": 5.846069812774658,
+      "learning_rate": 3.205128205128205e-06,
+      "loss": 8.8548,
+      "mean_token_accuracy": 0.2286823886005502,
+      "step": 20
+    },
+    {
+      "epoch": 0.00641333974667308,
+      "grad_norm": 6.527371406555176,
+      "learning_rate": 6.41025641025641e-06,
+      "loss": 8.6095,
+      "mean_token_accuracy": 0.236174263805151,
+      "step": 40
+    },
+    {
+      "epoch": 0.00962000962000962,
+      "grad_norm": 6.7473320960998535,
+      "learning_rate": 9.615384615384616e-06,
+      "loss": 7.0858,
+      "mean_token_accuracy": 0.24544834867119789,
+      "step": 60
+    },
+    {
+      "epoch": 0.01282667949334616,
+      "grad_norm": 3.6053383350372314,
+      "learning_rate": 1.282051282051282e-05,
+      "loss": 5.0951,
+      "mean_token_accuracy": 0.2442556582391262,
+      "step": 80
+    },
+    {
+      "epoch": 0.0160333493666827,
+      "grad_norm": 1.8141255378723145,
+      "learning_rate": 1.602564102564103e-05,
+      "loss": 4.0162,
+      "mean_token_accuracy": 0.29175431579351424,
+      "step": 100
+    },
+    {
+      "epoch": 0.01924001924001924,
+      "grad_norm": 1.9441587924957275,
+      "learning_rate": 1.923076923076923e-05,
+      "loss": 3.3901,
+      "mean_token_accuracy": 0.33477348536252977,
+      "step": 120
+    },
+    {
+      "epoch": 0.02244668911335578,
+      "grad_norm": 2.866448402404785,
+      "learning_rate": 2.2435897435897437e-05,
+      "loss": 3.0645,
+      "mean_token_accuracy": 0.36781402975320815,
+      "step": 140
+    },
+    {
+      "epoch": 0.02565335898669232,
+      "grad_norm": 6.173582553863525,
+      "learning_rate": 2.564102564102564e-05,
+      "loss": 2.8851,
+      "mean_token_accuracy": 0.3857575759291649,
+      "step": 160
+    },
+    {
+      "epoch": 0.02886002886002886,
+      "grad_norm": 2.632821798324585,
+      "learning_rate": 2.8846153846153845e-05,
+      "loss": 2.625,
+      "mean_token_accuracy": 0.42856554985046386,
+      "step": 180
+    },
+    {
+      "epoch": 0.0320666987333654,
+      "grad_norm": 3.801175594329834,
+      "learning_rate": 3.205128205128206e-05,
+      "loss": 2.5486,
+      "mean_token_accuracy": 0.44293054938316345,
+      "step": 200
+    },
+    {
+      "epoch": 0.03527336860670194,
+      "grad_norm": 2.8042311668395996,
+      "learning_rate": 3.525641025641026e-05,
+      "loss": 2.4267,
+      "mean_token_accuracy": 0.46162763684988023,
+      "step": 220
+    },
+    {
+      "epoch": 0.03848003848003848,
+      "grad_norm": 5.13622522354126,
+      "learning_rate": 3.846153846153846e-05,
+      "loss": 2.3457,
+      "mean_token_accuracy": 0.46204862743616104,
+      "step": 240
+    },
+    {
+      "epoch": 0.04168670835337502,
+      "grad_norm": 8.156232833862305,
+      "learning_rate": 4.166666666666667e-05,
+      "loss": 2.3276,
+      "mean_token_accuracy": 0.4628546267747879,
+      "step": 260
+    },
+    {
+      "epoch": 0.04489337822671156,
+      "grad_norm": 5.940677642822266,
+      "learning_rate": 4.4871794871794874e-05,
+      "loss": 2.3121,
+      "mean_token_accuracy": 0.4677517533302307,
+      "step": 280
+    },
+    {
+      "epoch": 0.0481000481000481,
+      "grad_norm": 3.2402215003967285,
+      "learning_rate": 4.8076923076923084e-05,
+      "loss": 2.2948,
+      "mean_token_accuracy": 0.4685308575630188,
+      "step": 300
+    },
+    {
+      "epoch": 0.05130671797338464,
+      "grad_norm": 4.654311656951904,
+      "learning_rate": 5.128205128205128e-05,
+      "loss": 2.2764,
+      "mean_token_accuracy": 0.46919394135475156,
+      "step": 320
+    },
+    {
+      "epoch": 0.05451338784672118,
+      "grad_norm": 6.181347370147705,
+      "learning_rate": 5.448717948717948e-05,
+      "loss": 2.2516,
+      "mean_token_accuracy": 0.4718445986509323,
+      "step": 340
+    },
+    {
+      "epoch": 0.05772005772005772,
+      "grad_norm": 4.977773189544678,
+      "learning_rate": 5.769230769230769e-05,
+      "loss": 2.3078,
+      "mean_token_accuracy": 0.46173652708530427,
+      "step": 360
+    },
+    {
+      "epoch": 0.06092672759339426,
+      "grad_norm": 2.649177074432373,
+      "learning_rate": 6.089743589743589e-05,
+      "loss": 2.2364,
+      "mean_token_accuracy": 0.4723318055272102,
+      "step": 380
+    },
+    {
+      "epoch": 0.0641333974667308,
+      "grad_norm": 2.475168228149414,
+      "learning_rate": 6.410256410256412e-05,
+      "loss": 2.2404,
+      "mean_token_accuracy": 0.4773345053195953,
+      "step": 400
+    },
+    {
+      "epoch": 0.06734006734006734,
+      "grad_norm": 3.376347541809082,
+      "learning_rate": 6.730769230769232e-05,
+      "loss": 2.1923,
+      "mean_token_accuracy": 0.48165750354528425,
+      "step": 420
+    },
+    {
+      "epoch": 0.07054673721340388,
+      "grad_norm": 2.1181466579437256,
+      "learning_rate": 7.051282051282052e-05,
+      "loss": 2.1944,
+      "mean_token_accuracy": 0.47836676687002183,
+      "step": 440
+    },
+    {
+      "epoch": 0.07375340708674041,
+      "grad_norm": 3.304994583129883,
+      "learning_rate": 7.371794871794872e-05,
+      "loss": 2.2055,
+      "mean_token_accuracy": 0.47685392051935194,
+      "step": 460
+    },
+    {
+      "epoch": 0.07696007696007696,
+      "grad_norm": 3.5151872634887695,
+      "learning_rate": 7.692307692307693e-05,
+      "loss": 2.171,
+      "mean_token_accuracy": 0.4798412054777145,
+      "step": 480
+    },
+    {
+      "epoch": 0.0801667468334135,
+      "grad_norm": 2.439455509185791,
+      "learning_rate": 8.012820512820514e-05,
+      "loss": 2.1592,
+      "mean_token_accuracy": 0.48496305793523786,
+      "step": 500
+    },
+    {
+      "epoch": 0.08337341670675004,
+      "grad_norm": 3.069302797317505,
+      "learning_rate": 8.333333333333334e-05,
+      "loss": 2.0884,
+      "mean_token_accuracy": 0.4890993982553482,
+      "step": 520
+    },
+    {
+      "epoch": 0.08658008658008658,
+      "grad_norm": 5.592751979827881,
+      "learning_rate": 8.653846153846155e-05,
+      "loss": 2.1425,
+      "mean_token_accuracy": 0.48478382378816604,
+      "step": 540
+    },
+    {
+      "epoch": 0.08978675645342311,
+      "grad_norm": 2.7817585468292236,
+      "learning_rate": 8.974358974358975e-05,
+      "loss": 2.1501,
+      "mean_token_accuracy": 0.4846224099397659,
+      "step": 560
+    },
+    {
+      "epoch": 0.09299342632675967,
+      "grad_norm": 3.0570271015167236,
+      "learning_rate": 9.294871794871795e-05,
+      "loss": 2.1297,
+      "mean_token_accuracy": 0.4879722028970718,
+      "step": 580
+    },
+    {
+      "epoch": 0.0962000962000962,
+      "grad_norm": 2.257469415664673,
+      "learning_rate": 9.615384615384617e-05,
+      "loss": 2.1088,
+      "mean_token_accuracy": 0.4889796242117882,
+      "step": 600
+    },
+    {
+      "epoch": 0.09940676607343274,
+      "grad_norm": 5.005424499511719,
+      "learning_rate": 9.935897435897437e-05,
+      "loss": 2.1248,
+      "mean_token_accuracy": 0.48695541769266126,
+      "step": 620
+    },
+    {
+      "epoch": 0.10261343594676928,
+      "grad_norm": 2.763110637664795,
+      "learning_rate": 9.999799512762136e-05,
+      "loss": 2.0852,
+      "mean_token_accuracy": 0.4945754811167717,
+      "step": 640
+    },
+    {
+      "epoch": 0.10582010582010581,
+      "grad_norm": 2.5508882999420166,
+      "learning_rate": 9.998985060913877e-05,
+      "loss": 2.1155,
+      "mean_token_accuracy": 0.48479859828948973,
+      "step": 660
+    },
+    {
+      "epoch": 0.10902677569344237,
+      "grad_norm": 1.9309931993484497,
+      "learning_rate": 9.99754421597847e-05,
+      "loss": 2.0774,
+      "mean_token_accuracy": 0.4935262441635132,
+      "step": 680
+    },
+    {
+      "epoch": 0.1122334455667789,
+      "grad_norm": 2.175483465194702,
+      "learning_rate": 9.995477158499624e-05,
+      "loss": 2.0698,
+      "mean_token_accuracy": 0.4930373579263687,
+      "step": 700
+    },
+    {
+      "epoch": 0.11544011544011544,
+      "grad_norm": 2.2829439640045166,
+      "learning_rate": 9.992784147488018e-05,
+      "loss": 2.064,
+      "mean_token_accuracy": 0.4933862343430519,
+      "step": 720
+    },
+    {
+      "epoch": 0.11864678531345198,
+      "grad_norm": 1.5809357166290283,
+      "learning_rate": 9.989465520388845e-05,
+      "loss": 2.0323,
+      "mean_token_accuracy": 0.5016711667180062,
+      "step": 740
+    },
+    {
+      "epoch": 0.12185345518678851,
+      "grad_norm": 2.230652332305908,
+      "learning_rate": 9.985521693039529e-05,
+      "loss": 2.075,
+      "mean_token_accuracy": 0.4931314527988434,
+      "step": 760
+    },
+    {
+      "epoch": 0.12506012506012507,
+      "grad_norm": 1.8354729413986206,
+      "learning_rate": 9.980953159617621e-05,
+      "loss": 2.0748,
+      "mean_token_accuracy": 0.4964547500014305,
+      "step": 780
+    },
+    {
+      "epoch": 0.1282667949334616,
+      "grad_norm": 2.15897274017334,
+      "learning_rate": 9.975760492578874e-05,
+      "loss": 2.0301,
+      "mean_token_accuracy": 0.4994760677218437,
+      "step": 800
+    },
+    {
+      "epoch": 0.13147346480679814,
+      "grad_norm": 2.6074705123901367,
+      "learning_rate": 9.969944342585508e-05,
+      "loss": 1.9838,
+      "mean_token_accuracy": 0.5075768753886223,
+      "step": 820
+    },
+    {
+      "epoch": 0.13468013468013468,
+      "grad_norm": 1.7504245042800903,
+      "learning_rate": 9.963505438424693e-05,
+      "loss": 2.0732,
+      "mean_token_accuracy": 0.49125437140464784,
+      "step": 840
+    },
+    {
+      "epoch": 0.13788680455347122,
+      "grad_norm": 2.214726686477661,
+      "learning_rate": 9.956444586917213e-05,
+      "loss": 2.0329,
+      "mean_token_accuracy": 0.4968407765030861,
+      "step": 860
+    },
+    {
+      "epoch": 0.14109347442680775,
+      "grad_norm": 1.6134241819381714,
+      "learning_rate": 9.948762672816377e-05,
+      "loss": 2.0392,
+      "mean_token_accuracy": 0.4989665150642395,
+      "step": 880
+    },
+    {
+      "epoch": 0.1443001443001443,
+      "grad_norm": 1.3124563694000244,
+      "learning_rate": 9.94046065869715e-05,
+      "loss": 2.0265,
+      "mean_token_accuracy": 0.5022040411829949,
+      "step": 900
+    },
+    {
+      "epoch": 0.14750681417348083,
+      "grad_norm": 2.182892084121704,
+      "learning_rate": 9.931539584835553e-05,
+      "loss": 2.0258,
+      "mean_token_accuracy": 0.4994161993265152,
+      "step": 920
+    },
+    {
+      "epoch": 0.1507134840468174,
+      "grad_norm": 2.158780813217163,
+      "learning_rate": 9.922000569078288e-05,
+      "loss": 2.038,
+      "mean_token_accuracy": 0.49309421628713607,
+      "step": 940
+    },
+    {
+      "epoch": 0.15392015392015393,
+      "grad_norm": 1.8986632823944092,
+      "learning_rate": 9.911844806702692e-05,
+      "loss": 2.0521,
+      "mean_token_accuracy": 0.4916775792837143,
+      "step": 960
+    },
+    {
+      "epoch": 0.15712682379349047,
+      "grad_norm": 2.2148079872131348,
+      "learning_rate": 9.901073570266946e-05,
+      "loss": 2.046,
+      "mean_token_accuracy": 0.4920832559466362,
+      "step": 980
+    },
+    {
+      "epoch": 0.160333493666827,
+      "grad_norm": 2.105100393295288,
+      "learning_rate": 9.889688209450627e-05,
+      "loss": 2.0154,
+      "mean_token_accuracy": 0.5017145529389382,
+      "step": 1000
+    },
+    {
+      "epoch": 0.16354016354016354,
+      "grad_norm": 1.9177722930908203,
+      "learning_rate": 9.877690150885588e-05,
+      "loss": 2.0133,
+      "mean_token_accuracy": 0.502875691652298,
+      "step": 1020
+    },
+    {
+      "epoch": 0.16674683341350008,
+      "grad_norm": 1.3427209854125977,
+      "learning_rate": 9.865080897977188e-05,
+      "loss": 1.9948,
+      "mean_token_accuracy": 0.505776722729206,
+      "step": 1040
+    },
+    {
+      "epoch": 0.16995350328683662,
+      "grad_norm": 2.0759928226470947,
+      "learning_rate": 9.851862030715919e-05,
+      "loss": 1.9401,
+      "mean_token_accuracy": 0.5120617598295212,
+      "step": 1060
+    },
+    {
+      "epoch": 0.17316017316017315,
+      "grad_norm": 1.2468688488006592,
+      "learning_rate": 9.838035205479419e-05,
+      "loss": 1.9913,
+      "mean_token_accuracy": 0.5054045274853707,
+      "step": 1080
+    },
+    {
+      "epoch": 0.1763668430335097,
+      "grad_norm": 1.2889540195465088,
+      "learning_rate": 9.823602154824927e-05,
+      "loss": 1.9963,
+      "mean_token_accuracy": 0.505834324657917,
+      "step": 1100
+    },
+    {
+      "epoch": 0.17957351290684623,
+      "grad_norm": 1.4190268516540527,
+      "learning_rate": 9.808564687272178e-05,
+      "loss": 1.9797,
+      "mean_token_accuracy": 0.5044485762715339,
+      "step": 1120
+    },
+    {
+      "epoch": 0.1827801827801828,
+      "grad_norm": 1.7106951475143433,
+      "learning_rate": 9.792924687076801e-05,
+      "loss": 1.9863,
+      "mean_token_accuracy": 0.5070938616991043,
+      "step": 1140
+    },
+    {
+      "epoch": 0.18598685265351933,
+      "grad_norm": 1.247165322303772,
+      "learning_rate": 9.776684113994204e-05,
+      "loss": 1.9723,
+      "mean_token_accuracy": 0.5093738153576851,
+      "step": 1160
+    },
+    {
+      "epoch": 0.18919352252685587,
+      "grad_norm": 1.4030295610427856,
+      "learning_rate": 9.759845003034007e-05,
+      "loss": 1.9756,
+      "mean_token_accuracy": 0.5056015223264694,
+      "step": 1180
+    },
+    {
+      "epoch": 0.1924001924001924,
+      "grad_norm": 1.243714690208435,
+      "learning_rate": 9.74240946420506e-05,
+      "loss": 1.9513,
+      "mean_token_accuracy": 0.5091609105467796,
+      "step": 1200
+    },
+    {
+      "epoch": 0.19560686227352894,
+      "grad_norm": 0.9765139222145081,
+      "learning_rate": 9.724379682251036e-05,
+      "loss": 1.9808,
+      "mean_token_accuracy": 0.50465277582407,
+      "step": 1220
+    },
+    {
+      "epoch": 0.19881353214686548,
+      "grad_norm": 1.4885112047195435,
+      "learning_rate": 9.705757916376682e-05,
+      "loss": 1.9993,
+      "mean_token_accuracy": 0.5045629054307937,
+      "step": 1240
+    },
+    {
+      "epoch": 0.20202020202020202,
+      "grad_norm": 1.258948802947998,
+      "learning_rate": 9.686546499964731e-05,
+      "loss": 1.9868,
+      "mean_token_accuracy": 0.5066259905695916,
+      "step": 1260
+    },
+    {
+      "epoch": 0.20522687189353855,
+      "grad_norm": 1.192187786102295,
+      "learning_rate": 9.666747840283512e-05,
+      "loss": 1.9444,
+      "mean_token_accuracy": 0.5078241482377053,
+      "step": 1280
+    },
+    {
+      "epoch": 0.2084335417668751,
+      "grad_norm": 1.1449286937713623,
+      "learning_rate": 9.646364418185325e-05,
+      "loss": 1.9964,
+      "mean_token_accuracy": 0.5036799103021622,
+      "step": 1300
+    },
+    {
+      "epoch": 0.21164021164021163,
+      "grad_norm": 1.2843917608261108,
+      "learning_rate": 9.62539878779556e-05,
+      "loss": 1.9857,
+      "mean_token_accuracy": 0.5052528932690621,
+      "step": 1320
+    },
+    {
+      "epoch": 0.2148468815135482,
+      "grad_norm": 1.1665757894515991,
+      "learning_rate": 9.603853576192674e-05,
+      "loss": 1.9676,
+      "mean_token_accuracy": 0.5088991582393646,
+      "step": 1340
+    },
+    {
+      "epoch": 0.21805355138688473,
+      "grad_norm": 1.0627950429916382,
+      "learning_rate": 9.581731483078995e-05,
+      "loss": 1.9494,
+      "mean_token_accuracy": 0.512041249871254,
+      "step": 1360
+    },
+    {
+      "epoch": 0.22126022126022127,
+      "grad_norm": 1.3954745531082153,
+      "learning_rate": 9.559035280442443e-05,
+      "loss": 1.9352,
+      "mean_token_accuracy": 0.5163529127836227,
+      "step": 1380
+    },
+    {
+      "epoch": 0.2244668911335578,
+      "grad_norm": 2.581529378890991,
+      "learning_rate": 9.535767812209185e-05,
+      "loss": 1.966,
+      "mean_token_accuracy": 0.5078106135129928,
+      "step": 1400
+    },
+    {
+      "epoch": 0.22767356100689434,
+      "grad_norm": 1.995743751525879,
+      "learning_rate": 9.511931993887289e-05,
+      "loss": 1.9278,
+      "mean_token_accuracy": 0.5141927003860474,
+      "step": 1420
+    },
+    {
+      "epoch": 0.23088023088023088,
+      "grad_norm": 1.3608646392822266,
+      "learning_rate": 9.487530812201384e-05,
+      "loss": 1.9464,
+      "mean_token_accuracy": 0.5119614064693451,
+      "step": 1440
+    },
+    {
+      "epoch": 0.23408690075356742,
+      "grad_norm": 2.1510233879089355,
+      "learning_rate": 9.462567324718428e-05,
+      "loss": 1.972,
+      "mean_token_accuracy": 0.5053435668349266,
+      "step": 1460
+    },
+    {
+      "epoch": 0.23729357062690395,
+      "grad_norm": 1.3489898443222046,
+      "learning_rate": 9.437044659464568e-05,
+      "loss": 1.9406,
+      "mean_token_accuracy": 0.5107260003685952,
+      "step": 1480
+    },
+    {
+      "epoch": 0.2405002405002405,
+      "grad_norm": 2.1510982513427734,
+      "learning_rate": 9.410966014533196e-05,
+      "loss": 1.9501,
+      "mean_token_accuracy": 0.5109146684408188,
+      "step": 1500
+    },
+    {
+      "epoch": 0.24370691037357703,
+      "grad_norm": 1.3027640581130981,
+      "learning_rate": 9.384334657684207e-05,
+      "loss": 1.9162,
+      "mean_token_accuracy": 0.5180920317769051,
+      "step": 1520
+    },
+    {
+      "epoch": 0.24691358024691357,
+      "grad_norm": 1.6839220523834229,
+      "learning_rate": 9.357153925934543e-05,
+      "loss": 1.932,
+      "mean_token_accuracy": 0.5122836321592331,
+      "step": 1540
+    },
+    {
+      "epoch": 0.25012025012025013,
+      "grad_norm": 1.5489161014556885,
+      "learning_rate": 9.329427225140042e-05,
+      "loss": 1.9287,
+      "mean_token_accuracy": 0.5156795844435692,
+      "step": 1560
+    },
+    {
+      "epoch": 0.25332691999358664,
+      "grad_norm": 1.1957494020462036,
+      "learning_rate": 9.301158029568684e-05,
+      "loss": 1.8905,
+      "mean_token_accuracy": 0.5207541212439537,
+      "step": 1580
+    },
+    {
+      "epoch": 0.2565335898669232,
+      "grad_norm": 1.2413902282714844,
+      "learning_rate": 9.272349881465236e-05,
+      "loss": 1.9309,
+      "mean_token_accuracy": 0.5141352891921998,
+      "step": 1600
+    },
+    {
+      "epoch": 0.2597402597402597,
+      "grad_norm": 1.3737406730651855,
+      "learning_rate": 9.243006390607403e-05,
+      "loss": 1.9565,
+      "mean_token_accuracy": 0.5097300469875335,
+      "step": 1620
+    },
+    {
+      "epoch": 0.2629469296135963,
+      "grad_norm": 1.24127197265625,
+      "learning_rate": 9.213131233853511e-05,
+      "loss": 1.945,
+      "mean_token_accuracy": 0.513680812716484,
+      "step": 1640
+    },
+    {
+      "epoch": 0.26615359948693285,
+      "grad_norm": 1.2227799892425537,
+      "learning_rate": 9.182728154681772e-05,
+      "loss": 1.945,
+      "mean_token_accuracy": 0.5159043714404106,
+      "step": 1660
+    },
+    {
+      "epoch": 0.26936026936026936,
+      "grad_norm": 1.4925428628921509,
+      "learning_rate": 9.151800962721218e-05,
+      "loss": 1.959,
+      "mean_token_accuracy": 0.5081851810216904,
+      "step": 1680
+    },
+    {
+      "epoch": 0.2725669392336059,
+      "grad_norm": 1.154524564743042,
+      "learning_rate": 9.12035353327434e-05,
+      "loss": 1.9162,
+      "mean_token_accuracy": 0.517485597729683,
+      "step": 1700
+    },
+    {
+      "epoch": 0.27577360910694243,
+      "grad_norm": 1.1241073608398438,
+      "learning_rate": 9.08838980683149e-05,
+      "loss": 1.9477,
+      "mean_token_accuracy": 0.5126137733459473,
+      "step": 1720
+    },
+    {
+      "epoch": 0.278980278980279,
+      "grad_norm": 1.8384186029434204,
+      "learning_rate": 9.055913788577128e-05,
+      "loss": 1.9197,
+      "mean_token_accuracy": 0.5140128180384635,
+      "step": 1740
+    },
+    {
+      "epoch": 0.2821869488536155,
+      "grad_norm": 1.3156044483184814,
+      "learning_rate": 9.02292954788796e-05,
+      "loss": 1.9079,
+      "mean_token_accuracy": 0.5170748963952064,
+      "step": 1760
+    },
+    {
+      "epoch": 0.28539361872695207,
+      "grad_norm": 1.449820876121521,
+      "learning_rate": 8.98944121782301e-05,
+      "loss": 1.9342,
+      "mean_token_accuracy": 0.5114595577120781,
+      "step": 1780
+    },
+    {
+      "epoch": 0.2886002886002886,
+      "grad_norm": 1.6988258361816406,
+      "learning_rate": 8.955452994605753e-05,
+      "loss": 1.9449,
+      "mean_token_accuracy": 0.5123026236891747,
+      "step": 1800
+    },
+    {
+      "epoch": 0.29180695847362514,
+      "grad_norm": 1.0350065231323242,
+      "learning_rate": 8.920969137098303e-05,
+      "loss": 1.9213,
+      "mean_token_accuracy": 0.5155784547328949,
+      "step": 1820
+    },
+    {
+      "epoch": 0.29501362834696165,
+      "grad_norm": 1.1465715169906616,
+      "learning_rate": 8.885993966267753e-05,
+      "loss": 1.8896,
+      "mean_token_accuracy": 0.5216770812869072,
+      "step": 1840
+    },
+    {
+      "epoch": 0.2982202982202982,
+      "grad_norm": 1.1059490442276,
+      "learning_rate": 8.850531864644749e-05,
+      "loss": 1.8978,
+      "mean_token_accuracy": 0.5205768331885338,
+      "step": 1860
+    },
+    {
+      "epoch": 0.3014269680936348,
+      "grad_norm": 1.199690818786621,
+      "learning_rate": 8.81458727577434e-05,
+      "loss": 1.9053,
+      "mean_token_accuracy": 0.5151138812303543,
+      "step": 1880
+    },
+    {
+      "epoch": 0.3046336379669713,
+      "grad_norm": 1.373304009437561,
+      "learning_rate": 8.77816470365918e-05,
+      "loss": 1.9276,
+      "mean_token_accuracy": 0.5131517440080643,
+      "step": 1900
+    },
+    {
+      "epoch": 0.30784030784030786,
+      "grad_norm": 1.4349356889724731,
+      "learning_rate": 8.741268712195166e-05,
+      "loss": 1.8921,
+      "mean_token_accuracy": 0.5225584238767624,
+      "step": 1920
+    },
+    {
+      "epoch": 0.31104697771364437,
+      "grad_norm": 1.0764795541763306,
+      "learning_rate": 8.703903924599548e-05,
+      "loss": 1.8639,
+      "mean_token_accuracy": 0.5256009042263031,
+      "step": 1940
+    },
+    {
+      "epoch": 0.31425364758698093,
+      "grad_norm": 1.946021318435669,
+      "learning_rate": 8.666075022831643e-05,
+      "loss": 1.881,
+      "mean_token_accuracy": 0.5257818832993507,
+      "step": 1960
+    },
+    {
+      "epoch": 0.31746031746031744,
+      "grad_norm": 1.2787318229675293,
+      "learning_rate": 8.627786747006145e-05,
+      "loss": 1.8966,
+      "mean_token_accuracy": 0.523352475464344,
+      "step": 1980
+    },
+    {
+      "epoch": 0.320666987333654,
+      "grad_norm": 1.7891284227371216,
+      "learning_rate": 8.589043894799185e-05,
+      "loss": 1.93,
+      "mean_token_accuracy": 0.5103022322058678,
+      "step": 2000
+    },
+    {
+      "epoch": 0.3238736572069905,
+      "grad_norm": 1.8391680717468262,
+      "learning_rate": 8.549851320847156e-05,
+      "loss": 1.8785,
+      "mean_token_accuracy": 0.5231836125254631,
+      "step": 2020
+    },
+    {
+      "epoch": 0.3270803270803271,
+      "grad_norm": 1.2025116682052612,
+      "learning_rate": 8.510213936138403e-05,
+      "loss": 1.8816,
+      "mean_token_accuracy": 0.5240169554948807,
+      "step": 2040
+    },
+    {
+      "epoch": 0.33028699695366365,
+      "grad_norm": 1.1192851066589355,
+      "learning_rate": 8.470136707397868e-05,
+      "loss": 1.8948,
+      "mean_token_accuracy": 0.5194600358605385,
+      "step": 2060
+    },
+    {
+      "epoch": 0.33349366682700016,
+      "grad_norm": 1.521708369255066,
+      "learning_rate": 8.42962465646473e-05,
+      "loss": 1.8973,
+      "mean_token_accuracy": 0.5208257511258125,
+      "step": 2080
+    },
+    {
+      "epoch": 0.3367003367003367,
+      "grad_norm": 1.6692718267440796,
+      "learning_rate": 8.388682859663152e-05,
+      "loss": 1.887,
+      "mean_token_accuracy": 0.5233332857489585,
+      "step": 2100
+    },
+    {
+      "epoch": 0.33990700657367323,
+      "grad_norm": 1.4780927896499634,
+      "learning_rate": 8.347316447166195e-05,
+      "loss": 1.8915,
+      "mean_token_accuracy": 0.523545055091381,
+      "step": 2120
+    },
+    {
+      "epoch": 0.3431136764470098,
+      "grad_norm": 1.7447547912597656,
+      "learning_rate": 8.305530602352985e-05,
+      "loss": 1.8826,
+      "mean_token_accuracy": 0.5198934078216553,
+      "step": 2140
+    },
+    {
+      "epoch": 0.3463203463203463,
+      "grad_norm": 1.1051859855651855,
+      "learning_rate": 8.263330561159222e-05,
+      "loss": 1.8792,
+      "mean_token_accuracy": 0.5219237297773361,
+      "step": 2160
+    },
+    {
+      "epoch": 0.34952701619368287,
+      "grad_norm": 1.2067945003509521,
+      "learning_rate": 8.220721611421087e-05,
+      "loss": 1.8989,
+      "mean_token_accuracy": 0.5167458400130271,
+      "step": 2180
+    },
+    {
+      "epoch": 0.3527336860670194,
+      "grad_norm": 1.3066425323486328,
+      "learning_rate": 8.177709092212663e-05,
+      "loss": 1.8421,
+      "mean_token_accuracy": 0.5322213441133499,
+      "step": 2200
+    },
+    {
+      "epoch": 0.35594035594035595,
+      "grad_norm": 1.0397696495056152,
+      "learning_rate": 8.134298393176916e-05,
+      "loss": 1.9187,
+      "mean_token_accuracy": 0.5156819596886635,
+      "step": 2220
+    },
+    {
+      "epoch": 0.35914702581369246,
+      "grad_norm": 1.3601312637329102,
+      "learning_rate": 8.090494953850365e-05,
+      "loss": 1.8785,
+      "mean_token_accuracy": 0.5235566765069961,
+      "step": 2240
+    },
+    {
+      "epoch": 0.362353695687029,
+      "grad_norm": 1.2964980602264404,
+      "learning_rate": 8.046304262981478e-05,
+      "loss": 1.8908,
+      "mean_token_accuracy": 0.5194398432970047,
+      "step": 2260
+    },
+    {
+      "epoch": 0.3655603655603656,
+      "grad_norm": 1.3470860719680786,
+      "learning_rate": 8.001731857842906e-05,
+      "loss": 1.9171,
+      "mean_token_accuracy": 0.51639633923769,
+      "step": 2280
+    },
+    {
+      "epoch": 0.3687670354337021,
+      "grad_norm": 1.3893637657165527,
+      "learning_rate": 7.956783323537652e-05,
+      "loss": 1.8682,
+      "mean_token_accuracy": 0.5249380469322205,
+      "step": 2300
+    },
+    {
+      "epoch": 0.37197370530703866,
+      "grad_norm": 1.5145325660705566,
+      "learning_rate": 7.911464292299222e-05,
+      "loss": 1.8457,
+      "mean_token_accuracy": 0.527319672703743,
+      "step": 2320
+    },
+    {
+      "epoch": 0.37518037518037517,
+      "grad_norm": 1.0269887447357178,
+      "learning_rate": 7.865780442785891e-05,
+      "loss": 1.8765,
+      "mean_token_accuracy": 0.5214496284723282,
+      "step": 2340
+    },
+    {
+      "epoch": 0.37838704505371173,
+      "grad_norm": 1.409474492073059,
+      "learning_rate": 7.81973749936914e-05,
+      "loss": 1.8571,
+      "mean_token_accuracy": 0.5259096622467041,
+      "step": 2360
+    },
+    {
+      "epoch": 0.38159371492704824,
+      "grad_norm": 1.0306053161621094,
+      "learning_rate": 7.77334123141638e-05,
+      "loss": 1.9028,
+      "mean_token_accuracy": 0.5194058448076249,
+      "step": 2380
+    },
+    {
+      "epoch": 0.3848003848003848,
+      "grad_norm": 1.076977014541626,
+      "learning_rate": 7.726597452568007e-05,
+      "loss": 1.8671,
+      "mean_token_accuracy": 0.5254542633891106,
+      "step": 2400
+    },
+    {
+      "epoch": 0.3880070546737213,
+      "grad_norm": 2.2276298999786377,
+      "learning_rate": 7.679512020008948e-05,
+      "loss": 1.8584,
+      "mean_token_accuracy": 0.5245337873697281,
+      "step": 2420
+    },
+    {
+      "epoch": 0.3912137245470579,
+      "grad_norm": 1.356619119644165,
+      "learning_rate": 7.632090833734723e-05,
+      "loss": 1.8601,
+      "mean_token_accuracy": 0.5262611448764801,
+      "step": 2440
+    },
+    {
+      "epoch": 0.3944203944203944,
+      "grad_norm": 1.5250309705734253,
+      "learning_rate": 7.584339835812151e-05,
+      "loss": 1.8671,
+      "mean_token_accuracy": 0.520966699719429,
+      "step": 2460
+    },
+    {
+      "epoch": 0.39762706429373096,
+      "grad_norm": 0.9932346940040588,
+      "learning_rate": 7.536265009634789e-05,
+      "loss": 1.8354,
+      "mean_token_accuracy": 0.5302907466888428,
+      "step": 2480
+    },
+    {
+      "epoch": 0.4008337341670675,
+      "grad_norm": 1.4524379968643188,
+      "learning_rate": 7.487872379173187e-05,
+      "loss": 1.8471,
+      "mean_token_accuracy": 0.527701361477375,
+      "step": 2500
+    },
+    {
+      "epoch": 0.40404040404040403,
+      "grad_norm": 1.6131216287612915,
+      "learning_rate": 7.439168008220057e-05,
+      "loss": 1.8791,
+      "mean_token_accuracy": 0.5199943497776985,
+      "step": 2520
+    },
+    {
+      "epoch": 0.4072470739137406,
+      "grad_norm": 1.2683628797531128,
+      "learning_rate": 7.390157999630463e-05,
+      "loss": 1.862,
+      "mean_token_accuracy": 0.5257236033678054,
+      "step": 2540
+    },
+    {
+      "epoch": 0.4104537437870771,
+      "grad_norm": 1.3786700963974,
+      "learning_rate": 7.340848494557098e-05,
+      "loss": 1.8329,
+      "mean_token_accuracy": 0.5322318792343139,
+      "step": 2560
+    },
+    {
+      "epoch": 0.4136604136604137,
+      "grad_norm": 1.2964134216308594,
+      "learning_rate": 7.291245671680781e-05,
+      "loss": 1.8378,
+      "mean_token_accuracy": 0.5295253932476044,
+      "step": 2580
+    },
+    {
+      "epoch": 0.4168670835337502,
+      "grad_norm": 1.2435472011566162,
+      "learning_rate": 7.241355746436243e-05,
+      "loss": 1.843,
+      "mean_token_accuracy": 0.528707692027092,
+      "step": 2600
+    },
+    {
+      "epoch": 0.42007375340708675,
+      "grad_norm": 1.6479510068893433,
+      "learning_rate": 7.191184970233299e-05,
+      "loss": 1.8559,
+      "mean_token_accuracy": 0.5243293821811676,
+      "step": 2620
+    },
+    {
+      "epoch": 0.42328042328042326,
+      "grad_norm": 1.4122897386550903,
+      "learning_rate": 7.140739629673531e-05,
+      "loss": 1.8584,
+      "mean_token_accuracy": 0.5285557866096496,
+      "step": 2640
+    },
+    {
+      "epoch": 0.4264870931537598,
+      "grad_norm": 1.2838547229766846,
+      "learning_rate": 7.09002604576255e-05,
+      "loss": 1.8609,
+      "mean_token_accuracy": 0.5250947296619415,
+      "step": 2660
+    },
+    {
+      "epoch": 0.4296937630270964,
+      "grad_norm": 1.5833513736724854,
+      "learning_rate": 7.03905057311794e-05,
+      "loss": 1.8246,
+      "mean_token_accuracy": 0.5320859342813492,
+      "step": 2680
+    },
+    {
+      "epoch": 0.4329004329004329,
+      "grad_norm": 1.5128042697906494,
+      "learning_rate": 6.987819599173007e-05,
+      "loss": 1.8332,
+      "mean_token_accuracy": 0.5332021147012711,
+      "step": 2700
+    },
+    {
+      "epoch": 0.43610710277376946,
+      "grad_norm": 1.0662071704864502,
+      "learning_rate": 6.936339543376411e-05,
+      "loss": 1.8757,
+      "mean_token_accuracy": 0.5246989518404007,
+      "step": 2720
+    },
+    {
+      "epoch": 0.43931377264710597,
+      "grad_norm": 1.0444306135177612,
+      "learning_rate": 6.884616856387768e-05,
+      "loss": 1.8411,
+      "mean_token_accuracy": 0.5272899508476258,
+      "step": 2740
+    },
+    {
+      "epoch": 0.44252044252044254,
+      "grad_norm": 1.4123362302780151,
+      "learning_rate": 6.832658019269374e-05,
+      "loss": 1.8165,
+      "mean_token_accuracy": 0.5287312120199203,
+      "step": 2760
+    },
+    {
+      "epoch": 0.44572711239377905,
+      "grad_norm": 1.2641295194625854,
+      "learning_rate": 6.780469542674088e-05,
+      "loss": 1.8581,
+      "mean_token_accuracy": 0.5240399688482285,
+      "step": 2780
+    },
+    {
+      "epoch": 0.4489337822671156,
+      "grad_norm": 1.055584192276001,
+      "learning_rate": 6.728057966029529e-05,
+      "loss": 1.7969,
+      "mean_token_accuracy": 0.536593297123909,
+      "step": 2800
+    },
+    {
+      "epoch": 0.4521404521404521,
+      "grad_norm": 1.2474592924118042,
+      "learning_rate": 6.675429856718652e-05,
+      "loss": 1.8295,
+      "mean_token_accuracy": 0.5313476771116257,
+      "step": 2820
+    },
+    {
+      "epoch": 0.4553471220137887,
+      "grad_norm": 1.1250567436218262,
+      "learning_rate": 6.62259180925684e-05,
+      "loss": 1.8102,
+      "mean_token_accuracy": 0.5341511443257332,
+      "step": 2840
+    },
+    {
+      "epoch": 0.4585537918871252,
+      "grad_norm": 1.2222751379013062,
+      "learning_rate": 6.569550444465568e-05,
+      "loss": 1.8343,
+      "mean_token_accuracy": 0.5306926101446152,
+      "step": 2860
+    },
+    {
+      "epoch": 0.46176046176046176,
+      "grad_norm": 1.24721097946167,
+      "learning_rate": 6.516312408642804e-05,
+      "loss": 1.8142,
+      "mean_token_accuracy": 0.5346602320671081,
+      "step": 2880
+    },
+    {
+      "epoch": 0.4649671316337983,
+      "grad_norm": 1.3725354671478271,
+      "learning_rate": 6.462884372730191e-05,
+      "loss": 1.8515,
+      "mean_token_accuracy": 0.525643666088581,
+      "step": 2900
+    },
+    {
+      "epoch": 0.46817380150713483,
+      "grad_norm": 1.5340793132781982,
+      "learning_rate": 6.409273031477152e-05,
+      "loss": 1.8405,
+      "mean_token_accuracy": 0.5296775609254837,
+      "step": 2920
+    },
+    {
+      "epoch": 0.4713804713804714,
+      "grad_norm": 1.4171345233917236,
+      "learning_rate": 6.355485102602012e-05,
+      "loss": 1.801,
+      "mean_token_accuracy": 0.5374062955379486,
+      "step": 2940
+    },
+    {
+      "epoch": 0.4745871412538079,
+      "grad_norm": 1.0726767778396606,
+      "learning_rate": 6.301527325950242e-05,
+      "loss": 1.8177,
+      "mean_token_accuracy": 0.5328266575932503,
+      "step": 2960
+    },
+    {
+      "epoch": 0.4777938111271445,
+      "grad_norm": 2.005058526992798,
+      "learning_rate": 6.247406462649927e-05,
+      "loss": 1.8109,
+      "mean_token_accuracy": 0.5314099788665771,
+      "step": 2980
+    },
+    {
+      "epoch": 0.481000481000481,
+      "grad_norm": 1.5543380975723267,
+      "learning_rate": 6.193129294264569e-05,
+      "loss": 1.8432,
+      "mean_token_accuracy": 0.5261458426713943,
+      "step": 3000
+    },
+    {
+      "epoch": 0.48420715087381755,
+      "grad_norm": 1.353722333908081,
+      "learning_rate": 6.138702621943334e-05,
+      "loss": 1.8178,
+      "mean_token_accuracy": 0.5276563599705696,
+      "step": 3020
+    },
+    {
+      "epoch": 0.48741382074715406,
+      "grad_norm": 1.1928757429122925,
+      "learning_rate": 6.0841332655688344e-05,
+      "loss": 1.8318,
+      "mean_token_accuracy": 0.5304355949163437,
+      "step": 3040
+    },
+    {
+      "epoch": 0.4906204906204906,
+      "grad_norm": 1.672419786453247,
+      "learning_rate": 6.029428062902571e-05,
+      "loss": 1.7971,
+      "mean_token_accuracy": 0.5380920082330704,
+      "step": 3060
+    },
+    {
+      "epoch": 0.49382716049382713,
+      "grad_norm": 1.231449842453003,
+      "learning_rate": 5.97459386872814e-05,
+      "loss": 1.8096,
+      "mean_token_accuracy": 0.5327497273683548,
+      "step": 3080
+    },
+    {
+      "epoch": 0.4970338303671637,
+      "grad_norm": 1.3776133060455322,
+      "learning_rate": 5.919637553992293e-05,
+      "loss": 1.8465,
+      "mean_token_accuracy": 0.5253860056400299,
+      "step": 3100
+    },
+    {
+      "epoch": 0.5000801667468334,
+      "eval_loss": 1.53940749168396,
+      "eval_mean_token_accuracy": 0.5933501680610359,
+      "eval_runtime": 1941.3293,
+      "eval_samples_per_second": 255.351,
+      "eval_steps_per_second": 1.995,
+      "step": 3119
+    },
+    {
+      "epoch": 0.5002405002405003,
+      "grad_norm": 1.2065333127975464,
+      "learning_rate": 5.864566004943983e-05,
+      "loss": 1.8275,
+      "mean_token_accuracy": 0.5308977961540222,
+      "step": 3120
+    },
+    {
+      "epoch": 0.5034471701138368,
+      "grad_norm": 1.3010640144348145,
+      "learning_rate": 5.8093861222714916e-05,
+      "loss": 1.8368,
+      "mean_token_accuracy": 0.5273123070597648,
+      "step": 3140
+    },
+    {
+      "epoch": 0.5066538399871733,
+      "grad_norm": 1.4349629878997803,
+      "learning_rate": 5.754104820237741e-05,
+      "loss": 1.8404,
+      "mean_token_accuracy": 0.5281692713499069,
+      "step": 3160
+    },
+    {
+      "epoch": 0.5098605098605099,
+      "grad_norm": 1.23130202293396,
+      "learning_rate": 5.698729025813908e-05,
+      "loss": 1.8004,
+      "mean_token_accuracy": 0.5355510145425797,
+      "step": 3180
+    },
+    {
+      "epoch": 0.5130671797338464,
+      "grad_norm": 1.785715937614441,
+      "learning_rate": 5.643265677811456e-05,
+      "loss": 1.8103,
+      "mean_token_accuracy": 0.5327317580580712,
+      "step": 3200
+    },
+    {
+      "epoch": 0.5162738496071829,
+      "grad_norm": 1.1478352546691895,
+      "learning_rate": 5.587721726012657e-05,
+      "loss": 1.7974,
+      "mean_token_accuracy": 0.5378333508968354,
+      "step": 3220
+    },
+    {
+      "epoch": 0.5194805194805194,
+      "grad_norm": 1.545850396156311,
+      "learning_rate": 5.532104130299771e-05,
+      "loss": 1.8375,
+      "mean_token_accuracy": 0.5303584784269333,
+      "step": 3240
+    },
+    {
+      "epoch": 0.522687189353856,
+      "grad_norm": 1.295379638671875,
+      "learning_rate": 5.476419859782942e-05,
+      "loss": 1.8123,
+      "mean_token_accuracy": 0.5323598384857178,
+      "step": 3260
+    },
+    {
+      "epoch": 0.5258938592271926,
+      "grad_norm": 1.2239121198654175,
+      "learning_rate": 5.4206758919269295e-05,
+      "loss": 1.7824,
+      "mean_token_accuracy": 0.5379300057888031,
+      "step": 3280
+    },
+    {
+      "epoch": 0.5291005291005291,
+      "grad_norm": 1.1906960010528564,
+      "learning_rate": 5.3648792116768166e-05,
+      "loss": 1.8102,
+      "mean_token_accuracy": 0.5297588974237442,
+      "step": 3300
+    },
+    {
+      "epoch": 0.5323071989738657,
+      "grad_norm": 1.3106406927108765,
+      "learning_rate": 5.3090368105827614e-05,
+      "loss": 1.8291,
+      "mean_token_accuracy": 0.5299944818019867,
+      "step": 3320
+    },
+    {
+      "epoch": 0.5355138688472022,
+      "grad_norm": 1.4072366952896118,
+      "learning_rate": 5.2531556859239305e-05,
+      "loss": 1.7814,
+      "mean_token_accuracy": 0.5385793894529343,
+      "step": 3340
+    },
+    {
+      "epoch": 0.5387205387205387,
+      "grad_norm": 1.286926031112671,
+      "learning_rate": 5.197242839831706e-05,
+      "loss": 1.8002,
+      "mean_token_accuracy": 0.5365041673183442,
+      "step": 3360
+    },
+    {
+      "epoch": 0.5419272085938752,
+      "grad_norm": 1.1429330110549927,
+      "learning_rate": 5.141305278412302e-05,
+      "loss": 1.7974,
+      "mean_token_accuracy": 0.5361256092786789,
+      "step": 3380
+    },
+    {
+      "epoch": 0.5451338784672118,
+      "grad_norm": 1.544176459312439,
+      "learning_rate": 5.085350010868855e-05,
+      "loss": 1.8282,
+      "mean_token_accuracy": 0.5302288249135018,
+      "step": 3400
+    },
+    {
+      "epoch": 0.5483405483405484,
+      "grad_norm": 1.5704865455627441,
+      "learning_rate": 5.029384048623154e-05,
+      "loss": 1.8004,
+      "mean_token_accuracy": 0.5345340296626091,
+      "step": 3420
+    },
+    {
+      "epoch": 0.5515472182138849,
+      "grad_norm": 1.841884970664978,
+      "learning_rate": 4.973414404437077e-05,
+      "loss": 1.8273,
+      "mean_token_accuracy": 0.527743574976921,
+      "step": 3440
+    },
+    {
+      "epoch": 0.5547538880872214,
+      "grad_norm": 1.3873976469039917,
+      "learning_rate": 4.917448091533865e-05,
+      "loss": 1.7882,
+      "mean_token_accuracy": 0.537655171751976,
+      "step": 3460
+    },
+    {
+      "epoch": 0.557960557960558,
+      "grad_norm": 1.487434983253479,
+      "learning_rate": 4.861492122719339e-05,
+      "loss": 1.836,
+      "mean_token_accuracy": 0.5291041225194931,
+      "step": 3480
+    },
+    {
+      "epoch": 0.5611672278338945,
+      "grad_norm": 1.2233349084854126,
+      "learning_rate": 4.805553509503154e-05,
+      "loss": 1.7568,
+      "mean_token_accuracy": 0.5442848384380341,
+      "step": 3500
+    },
+    {
+      "epoch": 0.564373897707231,
+      "grad_norm": 1.4371830224990845,
+      "learning_rate": 4.749639261220247e-05,
+      "loss": 1.7868,
+      "mean_token_accuracy": 0.5362420290708542,
+      "step": 3520
+    },
+    {
+      "epoch": 0.5675805675805676,
+      "grad_norm": 1.703787088394165,
+      "learning_rate": 4.6937563841525297e-05,
+      "loss": 1.8198,
+      "mean_token_accuracy": 0.5284188359975814,
+      "step": 3540
+    },
+    {
+      "epoch": 0.5707872374539041,
+      "grad_norm": 1.2655363082885742,
+      "learning_rate": 4.637911880650967e-05,
+      "loss": 1.8046,
+      "mean_token_accuracy": 0.533089104294777,
+      "step": 3560
+    },
+    {
+      "epoch": 0.5739939073272406,
+      "grad_norm": 1.2529635429382324,
+      "learning_rate": 4.582112748258166e-05,
+      "loss": 1.7947,
+      "mean_token_accuracy": 0.5354443103075027,
+      "step": 3580
+    },
+    {
+      "epoch": 0.5772005772005772,
+      "grad_norm": 1.6291149854660034,
+      "learning_rate": 4.526365978831552e-05,
+      "loss": 1.796,
+      "mean_token_accuracy": 0.5359267443418503,
+      "step": 3600
+    },
+    {
+      "epoch": 0.5804072470739138,
+      "grad_norm": 1.2783862352371216,
+      "learning_rate": 4.470678557667249e-05,
+      "loss": 1.8099,
+      "mean_token_accuracy": 0.5307971924543381,
+      "step": 3620
+    },
+    {
+      "epoch": 0.5836139169472503,
+      "grad_norm": 1.50846529006958,
+      "learning_rate": 4.415057462624812e-05,
+      "loss": 1.7749,
+      "mean_token_accuracy": 0.5401669204235077,
+      "step": 3640
+    },
+    {
+      "epoch": 0.5868205868205868,
+      "grad_norm": 1.3859307765960693,
+      "learning_rate": 4.3595096632528644e-05,
+      "loss": 1.7742,
+      "mean_token_accuracy": 0.539221653342247,
+      "step": 3660
+    },
+    {
+      "epoch": 0.5900272566939233,
+      "grad_norm": 1.4361897706985474,
+      "learning_rate": 4.304042119915776e-05,
+      "loss": 1.77,
+      "mean_token_accuracy": 0.5435908734798431,
+      "step": 3680
+    },
+    {
+      "epoch": 0.5932339265672599,
+      "grad_norm": 1.4237632751464844,
+      "learning_rate": 4.24866178292152e-05,
+      "loss": 1.7885,
+      "mean_token_accuracy": 0.5361862421035767,
+      "step": 3700
+    },
+    {
+      "epoch": 0.5964405964405964,
+      "grad_norm": 1.2441941499710083,
+      "learning_rate": 4.193375591650758e-05,
+      "loss": 1.7841,
+      "mean_token_accuracy": 0.5414396733045578,
+      "step": 3720
+    },
+    {
+      "epoch": 0.599647266313933,
+      "grad_norm": 1.211448073387146,
+      "learning_rate": 4.1381904736873075e-05,
+      "loss": 1.8035,
+      "mean_token_accuracy": 0.5305454939603805,
+      "step": 3740
+    },
+    {
+      "epoch": 0.6028539361872696,
+      "grad_norm": 1.489575982093811,
+      "learning_rate": 4.0831133439500954e-05,
+      "loss": 1.7602,
+      "mean_token_accuracy": 0.5400284379720688,
+      "step": 3760
+    },
+    {
+      "epoch": 0.6060606060606061,
+      "grad_norm": 1.2525299787521362,
+      "learning_rate": 4.0281511038266876e-05,
+      "loss": 1.7575,
+      "mean_token_accuracy": 0.5439072847366333,
+      "step": 3780
+    },
+    {
+      "epoch": 0.6092672759339426,
+      "grad_norm": 1.388637661933899,
+      "learning_rate": 3.9733106403085074e-05,
+      "loss": 1.764,
+      "mean_token_accuracy": 0.5406429708003998,
+      "step": 3800
+    },
+    {
+      "epoch": 0.6124739458072791,
+      "grad_norm": 1.2612781524658203,
+      "learning_rate": 3.918598825127885e-05,
+      "loss": 1.7477,
+      "mean_token_accuracy": 0.54590525329113,
+      "step": 3820
+    },
+    {
+      "epoch": 0.6156806156806157,
+      "grad_norm": 1.5898454189300537,
+      "learning_rate": 3.86402251389699e-05,
+      "loss": 1.7659,
+      "mean_token_accuracy": 0.541855725646019,
+      "step": 3840
+    },
+    {
+      "epoch": 0.6188872855539522,
+      "grad_norm": 1.2260034084320068,
+      "learning_rate": 3.809588545248794e-05,
+      "loss": 1.7809,
+      "mean_token_accuracy": 0.536695247888565,
+      "step": 3860
+    },
+    {
+      "epoch": 0.6220939554272887,
+      "grad_norm": 1.3900277614593506,
+      "learning_rate": 3.7553037399801705e-05,
+      "loss": 1.7894,
+      "mean_token_accuracy": 0.537989217042923,
+      "step": 3880
+    },
+    {
+      "epoch": 0.6253006253006252,
+      "grad_norm": 1.377323031425476,
+      "learning_rate": 3.701174900197218e-05,
+      "loss": 1.7522,
+      "mean_token_accuracy": 0.5395979732275009,
+      "step": 3900
+    },
+    {
+      "epoch": 0.6285072951739619,
+      "grad_norm": 1.9669792652130127,
+      "learning_rate": 3.647208808462921e-05,
+      "loss": 1.7819,
+      "mean_token_accuracy": 0.5357579261064529,
+      "step": 3920
+    },
+    {
+      "epoch": 0.6317139650472984,
+      "grad_norm": 1.3700226545333862,
+      "learning_rate": 3.5934122269472804e-05,
+      "loss": 1.7622,
+      "mean_token_accuracy": 0.5406556934118271,
+      "step": 3940
+    },
+    {
+      "epoch": 0.6349206349206349,
+      "grad_norm": 1.5546220541000366,
+      "learning_rate": 3.5397918965799785e-05,
+      "loss": 1.7627,
+      "mean_token_accuracy": 0.538935711979866,
+      "step": 3960
+    },
+    {
+      "epoch": 0.6381273047939715,
+      "grad_norm": 1.45100736618042,
+      "learning_rate": 3.486354536205712e-05,
+      "loss": 1.8306,
+      "mean_token_accuracy": 0.5290844351053238,
+      "step": 3980
+    },
+    {
+      "epoch": 0.641333974667308,
+      "grad_norm": 1.2580472230911255,
+      "learning_rate": 3.433106841742297e-05,
+      "loss": 1.783,
+      "mean_token_accuracy": 0.5352891981601715,
+      "step": 4000
+    },
+    {
+      "epoch": 0.6445406445406445,
+      "grad_norm": 1.4428609609603882,
+      "learning_rate": 3.380055485341644e-05,
+      "loss": 1.7703,
+      "mean_token_accuracy": 0.5361766919493676,
+      "step": 4020
+    },
+    {
+      "epoch": 0.647747314413981,
+      "grad_norm": 1.542856216430664,
+      "learning_rate": 3.3272071145537076e-05,
+      "loss": 1.7793,
+      "mean_token_accuracy": 0.532473573088646,
+      "step": 4040
+    },
+    {
+      "epoch": 0.6509539842873177,
+      "grad_norm": 1.307385802268982,
+      "learning_rate": 3.27456835149352e-05,
+      "loss": 1.7739,
+      "mean_token_accuracy": 0.5393928498029709,
+      "step": 4060
+    },
+    {
+      "epoch": 0.6541606541606542,
+      "grad_norm": 1.7225542068481445,
+      "learning_rate": 3.222145792011422e-05,
+      "loss": 1.7755,
+      "mean_token_accuracy": 0.5320593371987343,
+      "step": 4080
+    },
+    {
+      "epoch": 0.6573673240339907,
+      "grad_norm": 1.3483141660690308,
+      "learning_rate": 3.1699460048665594e-05,
+      "loss": 1.7686,
+      "mean_token_accuracy": 0.5363360077142716,
+      "step": 4100
+    },
+    {
+      "epoch": 0.6605739939073273,
+      "grad_norm": 1.4043909311294556,
+      "learning_rate": 3.117975530903813e-05,
+      "loss": 1.7762,
+      "mean_token_accuracy": 0.5329029619693756,
+      "step": 4120
+    },
+    {
+      "epoch": 0.6637806637806638,
+      "grad_norm": 1.434023380279541,
+      "learning_rate": 3.0662408822341864e-05,
+      "loss": 1.7682,
+      "mean_token_accuracy": 0.5406565859913826,
+      "step": 4140
+    },
+    {
+      "epoch": 0.6669873336540003,
+      "grad_norm": 1.3568503856658936,
+      "learning_rate": 3.014748541418817e-05,
+      "loss": 1.7466,
+      "mean_token_accuracy": 0.5411790430545806,
+      "step": 4160
+    },
+    {
+      "epoch": 0.6701940035273368,
+      "grad_norm": 1.4412897825241089,
+      "learning_rate": 2.9635049606566888e-05,
+      "loss": 1.7796,
+      "mean_token_accuracy": 0.5351216167211532,
+      "step": 4180
+    },
+    {
+      "epoch": 0.6734006734006734,
+      "grad_norm": 1.4259061813354492,
+      "learning_rate": 2.9125165609761463e-05,
+      "loss": 1.7679,
+      "mean_token_accuracy": 0.535513336956501,
+      "step": 4200
+    },
+    {
+      "epoch": 0.67660734327401,
+      "grad_norm": 1.517968773841858,
+      "learning_rate": 2.8617897314302994e-05,
+      "loss": 1.7773,
+      "mean_token_accuracy": 0.5354954585433006,
+      "step": 4220
+    },
+    {
+      "epoch": 0.6798140131473465,
+      "grad_norm": 1.5838121175765991,
+      "learning_rate": 2.811330828296471e-05,
+      "loss": 1.738,
+      "mean_token_accuracy": 0.545210886001587,
+      "step": 4240
+    },
+    {
+      "epoch": 0.683020683020683,
+      "grad_norm": 1.4874238967895508,
+      "learning_rate": 2.761146174279717e-05,
+      "loss": 1.8067,
+      "mean_token_accuracy": 0.5296819597482681,
+      "step": 4260
+    },
+    {
+      "epoch": 0.6862273528940196,
+      "grad_norm": 1.4804911613464355,
+      "learning_rate": 2.71124205772056e-05,
+      "loss": 1.798,
+      "mean_token_accuracy": 0.5329511255025864,
+      "step": 4280
+    },
+    {
+      "epoch": 0.6894340227673561,
+      "grad_norm": 1.4983785152435303,
+      "learning_rate": 2.6616247318070507e-05,
+      "loss": 1.7766,
+      "mean_token_accuracy": 0.536483982205391,
+      "step": 4300
+    },
+    {
+      "epoch": 0.6926406926406926,
+      "grad_norm": 1.352418065071106,
+      "learning_rate": 2.612300413791209e-05,
+      "loss": 1.749,
+      "mean_token_accuracy": 0.5425525337457657,
+      "step": 4320
+    },
+    {
+      "epoch": 0.6958473625140292,
+      "grad_norm": 1.5389214754104614,
+      "learning_rate": 2.5632752842099694e-05,
+      "loss": 1.7613,
+      "mean_token_accuracy": 0.5400553539395332,
+      "step": 4340
+    },
+    {
+      "epoch": 0.6990540323873657,
+      "grad_norm": 1.481913685798645,
+      "learning_rate": 2.5145554861107524e-05,
+      "loss": 1.7863,
+      "mean_token_accuracy": 0.5332476675510407,
+      "step": 4360
+    },
+    {
+      "epoch": 0.7022607022607023,
+      "grad_norm": 1.5351461172103882,
+      "learning_rate": 2.466147124281703e-05,
+      "loss": 1.7434,
+      "mean_token_accuracy": 0.5411887526512146,
+      "step": 4380
+    },
+    {
+      "epoch": 0.7054673721340388,
+      "grad_norm": 1.6004550457000732,
+      "learning_rate": 2.4180562644867406e-05,
+      "loss": 1.7678,
+      "mean_token_accuracy": 0.5376534581184387,
+      "step": 4400
+    },
+    {
+      "epoch": 0.7086740420073754,
+      "grad_norm": 1.4997788667678833,
+      "learning_rate": 2.3702889327054884e-05,
+      "loss": 1.7566,
+      "mean_token_accuracy": 0.5398379743099213,
+      "step": 4420
+    },
+    {
+      "epoch": 0.7118807118807119,
+      "grad_norm": 1.638317584991455,
+      "learning_rate": 2.3228511143782033e-05,
+      "loss": 1.7624,
+      "mean_token_accuracy": 0.5378725945949554,
+      "step": 4440
+    },
+    {
+      "epoch": 0.7150873817540484,
+      "grad_norm": 1.343430757522583,
+      "learning_rate": 2.2757487536557714e-05,
+      "loss": 1.751,
+      "mean_token_accuracy": 0.5400015458464622,
+      "step": 4460
+    },
+    {
+      "epoch": 0.7182940516273849,
+      "grad_norm": 1.2288793325424194,
+      "learning_rate": 2.2289877526548764e-05,
+      "loss": 1.727,
+      "mean_token_accuracy": 0.547047358751297,
+      "step": 4480
+    },
+    {
+      "epoch": 0.7215007215007215,
+      "grad_norm": 1.4691182374954224,
+      "learning_rate": 2.1825739707184494e-05,
+      "loss": 1.7667,
+      "mean_token_accuracy": 0.5362445533275604,
+      "step": 4500
+    },
+    {
+      "epoch": 0.724707391374058,
+      "grad_norm": 1.5017502307891846,
+      "learning_rate": 2.1365132236814668e-05,
+      "loss": 1.765,
+      "mean_token_accuracy": 0.5372408896684646,
+      "step": 4520
+    },
+    {
+      "epoch": 0.7279140612473946,
+      "grad_norm": 1.5174387693405151,
+      "learning_rate": 2.090811283142196e-05,
+      "loss": 1.7615,
+      "mean_token_accuracy": 0.5439996913075447,
+      "step": 4540
+    },
+    {
+      "epoch": 0.7311207311207312,
+      "grad_norm": 1.453343391418457,
+      "learning_rate": 2.0454738757390014e-05,
+      "loss": 1.7372,
+      "mean_token_accuracy": 0.540667149424553,
+      "step": 4560
+    },
+    {
+      "epoch": 0.7343274009940677,
+      "grad_norm": 1.578175663948059,
+      "learning_rate": 2.0005066824327673e-05,
+      "loss": 1.7301,
+      "mean_token_accuracy": 0.5443789809942245,
+      "step": 4580
+    },
+    {
+      "epoch": 0.7375340708674042,
+      "grad_norm": 1.8367490768432617,
+      "learning_rate": 1.9559153377950474e-05,
+      "loss": 1.7255,
+      "mean_token_accuracy": 0.5449193373322487,
+      "step": 4600
+    },
+    {
+      "epoch": 0.7407407407407407,
+      "grad_norm": 1.5051259994506836,
+      "learning_rate": 1.911705429302038e-05,
+      "loss": 1.7553,
+      "mean_token_accuracy": 0.5398281931877136,
+      "step": 4620
+    },
+    {
+      "epoch": 0.7439474106140773,
+      "grad_norm": 1.4641402959823608,
+      "learning_rate": 1.8678824966344416e-05,
+      "loss": 1.7288,
+      "mean_token_accuracy": 0.5467408001422882,
+      "step": 4640
+    },
+    {
+      "epoch": 0.7471540804874138,
+      "grad_norm": 1.3909623622894287,
+      "learning_rate": 1.824452030983316e-05,
+      "loss": 1.7688,
+      "mean_token_accuracy": 0.5377537578344345,
+      "step": 4660
+    },
+    {
+      "epoch": 0.7503607503607503,
+      "grad_norm": 1.541202187538147,
+      "learning_rate": 1.7814194743620172e-05,
+      "loss": 1.7575,
+      "mean_token_accuracy": 0.538825261592865,
+      "step": 4680
+    },
+    {
+      "epoch": 0.7535674202340868,
+      "grad_norm": 1.4056111574172974,
+      "learning_rate": 1.7387902189242826e-05,
+      "loss": 1.7503,
+      "mean_token_accuracy": 0.542974105477333,
+      "step": 4700
+    },
+    {
+      "epoch": 0.7567740901074235,
+      "grad_norm": 1.5985795259475708,
+      "learning_rate": 1.6965696062885815e-05,
+      "loss": 1.6876,
+      "mean_token_accuracy": 0.5525155961513519,
+      "step": 4720
+    },
+    {
+      "epoch": 0.75998075998076,
+      "grad_norm": 1.958160161972046,
+      "learning_rate": 1.6547629268687785e-05,
+      "loss": 1.7879,
+      "mean_token_accuracy": 0.5341216132044793,
+      "step": 4740
+    },
+    {
+      "epoch": 0.7631874298540965,
+      "grad_norm": 1.5259677171707153,
+      "learning_rate": 1.6133754192112353e-05,
+      "loss": 1.7234,
+      "mean_token_accuracy": 0.5443064093589782,
+      "step": 4760
+    },
+    {
+      "epoch": 0.7663940997274331,
+      "grad_norm": 1.8299003839492798,
+      "learning_rate": 1.572412269338386e-05,
+      "loss": 1.7196,
+      "mean_token_accuracy": 0.5485717713832855,
+      "step": 4780
+    },
+    {
+      "epoch": 0.7696007696007696,
+      "grad_norm": 1.6229146718978882,
+      "learning_rate": 1.5318786100989192e-05,
+      "loss": 1.7261,
+      "mean_token_accuracy": 0.5422624379396439,
+      "step": 4800
+    },
+    {
+      "epoch": 0.7728074394741061,
+      "grad_norm": 1.5998332500457764,
+      "learning_rate": 1.491779520524607e-05,
+      "loss": 1.7342,
+      "mean_token_accuracy": 0.5412924766540528,
+      "step": 4820
+    },
+    {
+      "epoch": 0.7760141093474426,
+      "grad_norm": 1.6178627014160156,
+      "learning_rate": 1.4521200251938777e-05,
+      "loss": 1.7244,
+      "mean_token_accuracy": 0.5459949374198914,
+      "step": 4840
+    },
+    {
+      "epoch": 0.7792207792207793,
+      "grad_norm": 1.718936800956726,
+      "learning_rate": 1.4129050936022215e-05,
+      "loss": 1.758,
+      "mean_token_accuracy": 0.5395539492368698,
+      "step": 4860
+    },
+    {
+      "epoch": 0.7824274490941158,
+      "grad_norm": 1.6473840475082397,
+      "learning_rate": 1.3741396395394918e-05,
+      "loss": 1.7264,
+      "mean_token_accuracy": 0.5445165723562241,
+      "step": 4880
+    },
+    {
+      "epoch": 0.7856341189674523,
+      "grad_norm": 1.7518523931503296,
+      "learning_rate": 1.3358285204741827e-05,
+      "loss": 1.7462,
+      "mean_token_accuracy": 0.5401648759841919,
+      "step": 4900
+    },
+    {
+      "epoch": 0.7888407888407888,
+      "grad_norm": 1.577514410018921,
+      "learning_rate": 1.2979765369447743e-05,
+      "loss": 1.7665,
+      "mean_token_accuracy": 0.5372074097394943,
+      "step": 4920
+    },
+    {
+      "epoch": 0.7920474587141254,
+      "grad_norm": 1.666795253753662,
+      "learning_rate": 1.2605884319582045e-05,
+      "loss": 1.7359,
+      "mean_token_accuracy": 0.54544777572155,
+      "step": 4940
+    },
+    {
+      "epoch": 0.7952541285874619,
+      "grad_norm": 1.4499692916870117,
+      "learning_rate": 1.2236688903955452e-05,
+      "loss": 1.7604,
+      "mean_token_accuracy": 0.5361103504896164,
+      "step": 4960
+    },
+    {
+      "epoch": 0.7984607984607984,
+      "grad_norm": 1.79007887840271,
+      "learning_rate": 1.1872225384249769e-05,
+      "loss": 1.734,
+      "mean_token_accuracy": 0.5440937787294388,
+      "step": 4980
+    },
+    {
+      "epoch": 0.801667468334135,
+      "grad_norm": 1.8798636198043823,
+      "learning_rate": 1.1512539429221058e-05,
+      "loss": 1.7989,
+      "mean_token_accuracy": 0.5301632598042488,
+      "step": 5000
+    },
+    {
+      "epoch": 0.8048741382074716,
+      "grad_norm": 1.6248602867126465,
+      "learning_rate": 1.1157676108977134e-05,
+      "loss": 1.7415,
+      "mean_token_accuracy": 0.5421714842319488,
+      "step": 5020
+    },
+    {
+      "epoch": 0.8080808080808081,
+      "grad_norm": 1.626051425933838,
+      "learning_rate": 1.0807679889330163e-05,
+      "loss": 1.7699,
+      "mean_token_accuracy": 0.5368034273386002,
+      "step": 5040
+    },
+    {
+      "epoch": 0.8112874779541446,
+      "grad_norm": 1.6709613800048828,
+      "learning_rate": 1.0462594626224886e-05,
+      "loss": 1.7285,
+      "mean_token_accuracy": 0.5429899528622627,
+      "step": 5060
+    },
+    {
+      "epoch": 0.8144941478274812,
+      "grad_norm": 1.5203421115875244,
+      "learning_rate": 1.012246356024325e-05,
+      "loss": 1.7242,
+      "mean_token_accuracy": 0.5429270610213279,
+      "step": 5080
+    },
+    {
+      "epoch": 0.8177008177008177,
+      "grad_norm": 1.6060646772384644,
+      "learning_rate": 9.78732931118625e-06,
+      "loss": 1.7546,
+      "mean_token_accuracy": 0.5418932989239693,
+      "step": 5100
+    },
+    {
+      "epoch": 0.8209074875741542,
+      "grad_norm": 1.7185138463974,
+      "learning_rate": 9.457233872733479e-06,
+      "loss": 1.7366,
+      "mean_token_accuracy": 0.5420786947011947,
+      "step": 5120
+    },
+    {
+      "epoch": 0.8241141574474908,
+      "grad_norm": 1.499786615371704,
+      "learning_rate": 9.132218607181109e-06,
+      "loss": 1.7197,
+      "mean_token_accuracy": 0.5424094408750534,
+      "step": 5140
+    },
+    {
+      "epoch": 0.8273208273208273,
+      "grad_norm": 1.7282419204711914,
+      "learning_rate": 8.812324240259096e-06,
+      "loss": 1.75,
+      "mean_token_accuracy": 0.5413718745112419,
+      "step": 5160
+    },
+    {
+      "epoch": 0.8305274971941639,
+      "grad_norm": 1.539486050605774,
+      "learning_rate": 8.497590856028026e-06,
+      "loss": 1.7148,
+      "mean_token_accuracy": 0.5459024518728256,
+      "step": 5180
+    },
+    {
+      "epoch": 0.8337341670675004,
+      "grad_norm": 1.6102161407470703,
+      "learning_rate": 8.188057891856426e-06,
+      "loss": 1.7215,
+      "mean_token_accuracy": 0.544721120595932,
+      "step": 5200
+    },
+    {
+      "epoch": 0.836940836940837,
+      "grad_norm": 1.4731084108352661,
+      "learning_rate": 7.883764133479139e-06,
+      "loss": 1.7334,
+      "mean_token_accuracy": 0.5424891680479049,
+      "step": 5220
+    },
+    {
+      "epoch": 0.8401475068141735,
+      "grad_norm": 1.6069941520690918,
+      "learning_rate": 7.5847477101372855e-06,
+      "loss": 1.7146,
+      "mean_token_accuracy": 0.5504184126853943,
+      "step": 5240
+    },
+    {
+      "epoch": 0.84335417668751,
+      "grad_norm": 1.6103508472442627,
+      "learning_rate": 7.291046089800496e-06,
+      "loss": 1.7645,
+      "mean_token_accuracy": 0.5386257976293564,
+      "step": 5260
+    },
+    {
+      "epoch": 0.8465608465608465,
+      "grad_norm": 1.575320839881897,
+      "learning_rate": 7.002696074472076e-06,
+      "loss": 1.7392,
+      "mean_token_accuracy": 0.5386416122317315,
+      "step": 5280
+    },
+    {
+      "epoch": 0.8497675164341831,
+      "grad_norm": 1.6121926307678223,
+      "learning_rate": 6.719733795577504e-06,
+      "loss": 1.7426,
+      "mean_token_accuracy": 0.5400305777788162,
+      "step": 5300
+    },
+    {
+      "epoch": 0.8529741863075196,
+      "grad_norm": 1.8504875898361206,
+      "learning_rate": 6.442194709437089e-06,
+      "loss": 1.7193,
+      "mean_token_accuracy": 0.546892175078392,
+      "step": 5320
+    },
+    {
+      "epoch": 0.8561808561808562,
+      "grad_norm": 1.537227988243103,
+      "learning_rate": 6.170113592823057e-06,
+      "loss": 1.7108,
+      "mean_token_accuracy": 0.5477687418460846,
+      "step": 5340
+    },
+    {
+      "epoch": 0.8593875260541928,
+      "grad_norm": 1.7463566064834595,
+      "learning_rate": 5.90352453860199e-06,
+      "loss": 1.7716,
+      "mean_token_accuracy": 0.5326795309782029,
+      "step": 5360
+    },
+    {
+      "epoch": 0.8625941959275293,
+      "grad_norm": 1.628735065460205,
+      "learning_rate": 5.642460951462747e-06,
+      "loss": 1.7015,
+      "mean_token_accuracy": 0.5511958628892899,
+      "step": 5380
+    },
+    {
+      "epoch": 0.8658008658008658,
+      "grad_norm": 1.5532407760620117,
+      "learning_rate": 5.3869555437307985e-06,
+      "loss": 1.7055,
+      "mean_token_accuracy": 0.5501611769199372,
+      "step": 5400
+    },
+    {
+      "epoch": 0.8690075356742023,
+      "grad_norm": 1.3285646438598633,
+      "learning_rate": 5.137040331269211e-06,
+      "loss": 1.6782,
+      "mean_token_accuracy": 0.5495849221944809,
+      "step": 5420
+    },
+    {
+      "epoch": 0.8722142055475389,
+      "grad_norm": 1.8813681602478027,
+      "learning_rate": 4.892746629466899e-06,
+      "loss": 1.7268,
+      "mean_token_accuracy": 0.5474593430757523,
+      "step": 5440
+    },
+    {
+      "epoch": 0.8754208754208754,
+      "grad_norm": 1.5516417026519775,
+      "learning_rate": 4.654105049314744e-06,
+      "loss": 1.7426,
+      "mean_token_accuracy": 0.544597877562046,
+      "step": 5460
+    },
+    {
+      "epoch": 0.8786275452942119,
+      "grad_norm": 1.6854817867279053,
+      "learning_rate": 4.421145493569861e-06,
+      "loss": 1.7039,
+      "mean_token_accuracy": 0.5486319988965989,
+      "step": 5480
+    },
+    {
+      "epoch": 0.8818342151675485,
+      "grad_norm": 1.6205663681030273,
+      "learning_rate": 4.19389715300868e-06,
+      "loss": 1.708,
+      "mean_token_accuracy": 0.5509619444608689,
+      "step": 5500
+    },
+    {
+      "epoch": 0.8850408850408851,
+      "grad_norm": 1.7468833923339844,
+      "learning_rate": 3.972388502769226e-06,
+      "loss": 1.7353,
+      "mean_token_accuracy": 0.5436493963003158,
+      "step": 5520
+    },
+    {
+      "epoch": 0.8882475549142216,
+      "grad_norm": 1.6175867319107056,
+      "learning_rate": 3.7566472987830726e-06,
+      "loss": 1.7306,
+      "mean_token_accuracy": 0.5433537870645523,
+      "step": 5540
+    },
+    {
+      "epoch": 0.8914542247875581,
+      "grad_norm": 1.6812947988510132,
+      "learning_rate": 3.546700574297385e-06,
+      "loss": 1.7391,
+      "mean_token_accuracy": 0.5441039055585861,
+      "step": 5560
+    },
+    {
+      "epoch": 0.8946608946608947,
+      "grad_norm": 1.6716430187225342,
+      "learning_rate": 3.3425746364875834e-06,
+      "loss": 1.7076,
+      "mean_token_accuracy": 0.5486252635717392,
+      "step": 5580
+    },
+    {
+      "epoch": 0.8978675645342312,
+      "grad_norm": 1.838515281677246,
+      "learning_rate": 3.144295063160929e-06,
+      "loss": 1.7275,
+      "mean_token_accuracy": 0.5441503524780273,
+      "step": 5600
+    },
+    {
+      "epoch": 0.9010742344075677,
+      "grad_norm": 1.9950391054153442,
+      "learning_rate": 2.9518866995515038e-06,
+      "loss": 1.713,
+      "mean_token_accuracy": 0.5466269612312317,
+      "step": 5620
+    },
+    {
+      "epoch": 0.9042809042809042,
+      "grad_norm": 1.6081202030181885,
+      "learning_rate": 2.765373655207021e-06,
+      "loss": 1.7294,
+      "mean_token_accuracy": 0.5423602730035781,
+      "step": 5640
+    },
+    {
+      "epoch": 0.9074875741542409,
+      "grad_norm": 1.6213099956512451,
+      "learning_rate": 2.5847793009678212e-06,
+      "loss": 1.723,
+      "mean_token_accuracy": 0.5430870592594147,
+      "step": 5660
+    },
+    {
+      "epoch": 0.9106942440275774,
+      "grad_norm": 1.8376996517181396,
+      "learning_rate": 2.4101262660383594e-06,
+      "loss": 1.7215,
+      "mean_token_accuracy": 0.5455273285508155,
+      "step": 5680
+    },
+    {
+      "epoch": 0.9139009139009139,
+      "grad_norm": 1.6142514944076538,
+      "learning_rate": 2.241436435151717e-06,
+      "loss": 1.7415,
+      "mean_token_accuracy": 0.5425774544477463,
+      "step": 5700
+    },
+    {
+      "epoch": 0.9171075837742504,
+      "grad_norm": 2.0796701908111572,
+      "learning_rate": 2.078730945827351e-06,
+      "loss": 1.7411,
+      "mean_token_accuracy": 0.5402660548686982,
+      "step": 5720
+    },
+    {
+      "epoch": 0.920314253647587,
+      "grad_norm": 1.64828622341156,
+      "learning_rate": 1.9220301857224455e-06,
+      "loss": 1.7002,
+      "mean_token_accuracy": 0.5491351515054703,
+      "step": 5740
+    },
+    {
+      "epoch": 0.9235209235209235,
+      "grad_norm": 1.402880311012268,
+      "learning_rate": 1.7713537900772958e-06,
+      "loss": 1.7124,
+      "mean_token_accuracy": 0.5458125352859498,
+      "step": 5760
+    },
+    {
+      "epoch": 0.92672759339426,
+      "grad_norm": 1.4213367700576782,
+      "learning_rate": 1.626720639254914e-06,
+      "loss": 1.6927,
+      "mean_token_accuracy": 0.5488159418106079,
+      "step": 5780
+    },
+    {
+      "epoch": 0.9299342632675967,
+      "grad_norm": 1.6133363246917725,
+      "learning_rate": 1.4881488563752377e-06,
+      "loss": 1.7388,
+      "mean_token_accuracy": 0.5443336263298988,
+      "step": 5800
+    },
+    {
+      "epoch": 0.9331409331409332,
+      "grad_norm": 1.4495124816894531,
+      "learning_rate": 1.3556558050442425e-06,
+      "loss": 1.704,
+      "mean_token_accuracy": 0.546082928776741,
+      "step": 5820
+    },
+    {
+      "epoch": 0.9363476030142697,
+      "grad_norm": 1.6041686534881592,
+      "learning_rate": 1.2292580871782144e-06,
+      "loss": 1.7317,
+      "mean_token_accuracy": 0.5426234900951385,
+      "step": 5840
+    },
+    {
+      "epoch": 0.9395542728876062,
+      "grad_norm": 1.742559790611267,
+      "learning_rate": 1.1089715409234313e-06,
+      "loss": 1.7058,
+      "mean_token_accuracy": 0.547260707616806,
+      "step": 5860
+    },
+    {
+      "epoch": 0.9427609427609428,
+      "grad_norm": 1.509397029876709,
+      "learning_rate": 9.948112386716168e-07,
+      "loss": 1.6986,
+      "mean_token_accuracy": 0.5505899757146835,
+      "step": 5880
+    },
+    {
+      "epoch": 0.9459676126342793,
+      "grad_norm": 1.6833570003509521,
+      "learning_rate": 8.86791485171301e-07,
+      "loss": 1.7469,
+      "mean_token_accuracy": 0.5377726525068283,
+      "step": 5900
+    },
+    {
+      "epoch": 0.9491742825076158,
+      "grad_norm": 1.5239343643188477,
+      "learning_rate": 7.849258157353546e-07,
+      "loss": 1.7162,
+      "mean_token_accuracy": 0.5405813604593277,
+      "step": 5920
+    },
+    {
+      "epoch": 0.9523809523809523,
+      "grad_norm": 1.7833971977233887,
+      "learning_rate": 6.892269945449781e-07,
+      "loss": 1.7317,
+      "mean_token_accuracy": 0.545667690038681,
+      "step": 5940
+    },
+    {
+      "epoch": 0.955587622254289,
+      "grad_norm": 1.5971351861953735,
+      "learning_rate": 5.997070130503101e-07,
+      "loss": 1.7468,
+      "mean_token_accuracy": 0.5399376690387726,
+      "step": 5960
+    },
+    {
+      "epoch": 0.9587942921276255,
+      "grad_norm": 1.791557788848877,
+      "learning_rate": 5.163770884678065e-07,
+      "loss": 1.7308,
+      "mean_token_accuracy": 0.5437323659658432,
+      "step": 5980
+    },
+    {
+      "epoch": 0.962000962000962,
+      "grad_norm": 1.9151041507720947,
+      "learning_rate": 4.392476623747366e-07,
+      "loss": 1.741,
+      "mean_token_accuracy": 0.5398768007755279,
+      "step": 6000
+    },
+    {
+      "epoch": 0.9652076318742986,
+      "grad_norm": 1.6582918167114258,
+      "learning_rate": 3.6832839940075805e-07,
+      "loss": 1.7006,
+      "mean_token_accuracy": 0.5496091902256012,
+      "step": 6020
+    },
+    {
+      "epoch": 0.9684143017476351,
+      "grad_norm": 1.8586136102676392,
+      "learning_rate": 3.0362818601693543e-07,
+      "loss": 1.7186,
+      "mean_token_accuracy": 0.54738067984581,
+      "step": 6040
+    },
+    {
+      "epoch": 0.9716209716209716,
+      "grad_norm": 1.8676416873931885,
+      "learning_rate": 2.451551294222087e-07,
+      "loss": 1.7165,
+      "mean_token_accuracy": 0.5441298902034759,
+      "step": 6060
+    },
+    {
+      "epoch": 0.9748276414943081,
+      "grad_norm": 1.6972177028656006,
+      "learning_rate": 1.929165565275448e-07,
+      "loss": 1.7179,
+      "mean_token_accuracy": 0.543172636628151,
+      "step": 6080
+    },
+    {
+      "epoch": 0.9780343113676447,
+      "grad_norm": 1.5021824836730957,
+      "learning_rate": 1.4691901303782173e-07,
+      "loss": 1.7311,
+      "mean_token_accuracy": 0.5438661694526672,
+      "step": 6100
+    },
+    {
+      "epoch": 0.9812409812409812,
+      "grad_norm": 1.6856005191802979,
+      "learning_rate": 1.0716826263165725e-07,
+      "loss": 1.7283,
+      "mean_token_accuracy": 0.5454312190413475,
+      "step": 6120
+    },
+    {
+      "epoch": 0.9844476511143178,
+      "grad_norm": 1.5837814807891846,
+      "learning_rate": 7.366928623916413e-08,
+      "loss": 1.7156,
+      "mean_token_accuracy": 0.5453536838293076,
+      "step": 6140
+    },
+    {
+      "epoch": 0.9876543209876543,
+      "grad_norm": 1.9022527933120728,
+      "learning_rate": 4.64262814178551e-08,
+      "loss": 1.7011,
+      "mean_token_accuracy": 0.5509391695261001,
+      "step": 6160
+    },
+    {
+      "epoch": 0.9908609908609909,
+      "grad_norm": 1.4848562479019165,
+      "learning_rate": 2.544266182662458e-08,
+      "loss": 1.6851,
+      "mean_token_accuracy": 0.5534208327531814,
+      "step": 6180
+    },
+    {
+      "epoch": 0.9940676607343274,
+      "grad_norm": 1.5964195728302002,
+      "learning_rate": 1.0721056798057572e-08,
+      "loss": 1.7122,
+      "mean_token_accuracy": 0.5429927557706833,
+      "step": 6200
+    },
+    {
+      "epoch": 0.9972743306076639,
+      "grad_norm": 1.8827787637710571,
+      "learning_rate": 2.2633110089265163e-09,
+      "loss": 1.7211,
+      "mean_token_accuracy": 0.5458635985851288,
+      "step": 6220
+    }
+  ],
+  "logging_steps": 20,
+  "max_steps": 6237,
+  "num_input_tokens_seen": 0,
+  "num_train_epochs": 1,
+  "save_steps": 3119,
+  "stateful_callbacks": {
+    "TrainerControl": {
+      "args": {
+        "should_epoch_stop": false,
+        "should_evaluate": false,
+        "should_log": false,
+        "should_save": true,
+        "should_training_stop": true
+      },
+      "attributes": {}
+    }
+  },
+  "total_flos": 3.9136612672224625e+18,
+  "train_batch_size": 16,
+  "trial_name": null,
+  "trial_params": null
+}

checkpoint-6237/training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:aad656740548acb2cf8355c0bb58f2d3acfb74bed369d470a571070c08698a6c
+size 5880

config.json

@@ -0,0 +1,55 @@
+{
+  "_attn_implementation_autoset": true,
+  "_name_or_path": "google/paligemma2-3b-pt-224",
+  "_vocab_size": 257152,
+  "architectures": [
+    "PaliGemmaForConditionalGeneration"
+  ],
+  "bos_token_id": 2,
+  "eos_token_id": 1,
+  "hidden_size": 2048,
+  "image_token_index": 257152,
+  "model_type": "paligemma",
+  "num_hidden_layers": 26,
+  "pad_token_id": 0,
+  "projection_dim": 2304,
+  "text_config": {
+    "architectures": [
+      "Gemma2ForCausalLM"
+    ],
+    "attn_logit_softcapping": 50.0,
+    "cache_implementation": "hybrid",
+    "eos_token_id": [
+      1,
+      107
+    ],
+    "final_logit_softcapping": 30.0,
+    "hidden_act": "gelu_pytorch_tanh",
+    "hidden_activation": "gelu_pytorch_tanh",
+    "hidden_size": 2304,
+    "intermediate_size": 9216,
+    "model_type": "gemma2",
+    "num_hidden_layers": 26,
+    "num_image_tokens": 256,
+    "num_key_value_heads": 4,
+    "query_pre_attn_scalar": 256,
+    "sliding_window": 4096,
+    "torch_dtype": "bfloat16",
+    "vocab_size": 257216
+  },
+  "torch_dtype": "bfloat16",
+  "transformers_version": "4.49.0",
+  "vision_config": {
+    "hidden_size": 1152,
+    "intermediate_size": 4304,
+    "model_type": "siglip_vision_model",
+    "num_attention_heads": 16,
+    "num_hidden_layers": 27,
+    "num_image_tokens": 256,
+    "num_positions": 256,
+    "patch_size": 14,
+    "projection_dim": 2304,
+    "torch_dtype": "bfloat16",
+    "vision_use_head": false
+  }
+}

eval_results.json

@@ -0,0 +1,7 @@
+{
+    "eval_loss": 1.53940749168396,
+    "eval_runtime": 1828.6515,
+    "eval_samples": 495720,
+    "eval_samples_per_second": 271.085,
+    "eval_steps_per_second": 2.118
+}

preprocessor_config.json

@@ -0,0 +1,25 @@
+{
+  "do_convert_rgb": null,
+  "do_normalize": true,
+  "do_rescale": true,
+  "do_resize": true,
+  "image_mean": [
+    0.5,
+    0.5,
+    0.5
+  ],
+  "image_processor_type": "SiglipImageProcessor",
+  "image_seq_length": 256,
+  "image_std": [
+    0.5,
+    0.5,
+    0.5
+  ],
+  "processor_class": "PaliGemmaProcessor",
+  "resample": 3,
+  "rescale_factor": 0.00392156862745098,
+  "size": {
+    "height": 224,
+    "width": 224
+  }
+}

special_tokens_map.json

@@ -0,0 +1,39 @@
+{
+  "additional_special_tokens": [
+    {
+      "content": "<image>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false
+    }
+  ],
+  "bos_token": {
+    "content": "<bos>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "<eos>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "<pad>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "unk_token": {
+    "content": "<unk>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

tokenizer.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b648d11e0879b11659e6b4051f691752c0cef597a865c6fde5b318b9f68c1d05
+size 34600974

train_results.json

@@ -0,0 +1,8 @@
+{
+    "total_flos": 3.9136612672224625e+18,
+    "train_loss": 1.9499194212634154,
+    "train_runtime": 7999.3345,
+    "train_samples": 798264,
+    "train_samples_per_second": 99.791,
+    "train_steps_per_second": 0.78
+}