Upload folder using huggingface_hub

.gitattributes

@@ -70,3 +70,8 @@ experiments/de-pt-zh/adapter/tokenizer.json filter=lfs diff=lfs merge=lfs -text
 experiments/de-pt-zh/checkpoint-1000/tokenizer.json filter=lfs diff=lfs merge=lfs -text
 experiments/de-pt-zh/checkpoint-1185/tokenizer.json filter=lfs diff=lfs merge=lfs -text
 experiments/de-pt-zh/checkpoint-500/tokenizer.json filter=lfs diff=lfs merge=lfs -text
+experiments/de-en-pt-zh/adapter/tokenizer.json filter=lfs diff=lfs merge=lfs -text
+experiments/de-en-pt-zh/checkpoint-1000/tokenizer.json filter=lfs diff=lfs merge=lfs -text
+experiments/de-en-pt-zh/checkpoint-1500/tokenizer.json filter=lfs diff=lfs merge=lfs -text
+experiments/de-en-pt-zh/checkpoint-1580/tokenizer.json filter=lfs diff=lfs merge=lfs -text
+experiments/de-en-pt-zh/checkpoint-500/tokenizer.json filter=lfs diff=lfs merge=lfs -text

experiments/de-en-pt-zh/README.md

@@ -0,0 +1,59 @@
+---
+base_model: unsloth/meta-llama-3.1-8b-instruct-bnb-4bit
+library_name: transformers
+model_name: de-en-pt-zh
+tags:
+- generated_from_trainer
+- unsloth
+- sft
+- trl
+licence: license
+---
+
+# Model Card for de-en-pt-zh
+
+This model is a fine-tuned version of [unsloth/meta-llama-3.1-8b-instruct-bnb-4bit](https://huggingface.co/unsloth/meta-llama-3.1-8b-instruct-bnb-4bit).
+It has been trained using [TRL](https://github.com/huggingface/trl).
+
+## Quick start
+
+```python
+from transformers import pipeline
+
+question = "If you had a time machine, but could only go to the past or the future once and never return, which would you choose and why?"
+generator = pipeline("text-generation", model="None", device="cuda")
+output = generator([{"role": "user", "content": question}], max_new_tokens=128, return_full_text=False)[0]
+print(output["generated_text"])
+```
+
+## Training procedure
+
+ 
+
+
+This model was trained with SFT.
+
+### Framework versions
+
+- TRL: 0.22.2
+- Transformers: 4.56.0
+- Pytorch: 2.8.0+cu129
+- Datasets: 4.3.0
+- Tokenizers: 0.22.0
+
+## Citations
+
+
+
+Cite TRL as:
+    
+```bibtex
+@misc{vonwerra2022trl,
+	title        = {{TRL: Transformer Reinforcement Learning}},
+	author       = {Leandro von Werra and Younes Belkada and Lewis Tunstall and Edward Beeching and Tristan Thrush and Nathan Lambert and Shengyi Huang and Kashif Rasul and Quentin Gallou{\'e}dec},
+	year         = 2020,
+	journal      = {GitHub repository},
+	publisher    = {GitHub},
+	howpublished = {\url{https://github.com/huggingface/trl}}
+}
+```

experiments/de-en-pt-zh/adapter/README.md

@@ -0,0 +1,210 @@
+---
+base_model: unsloth/meta-llama-3.1-8b-instruct-bnb-4bit
+library_name: peft
+pipeline_tag: text-generation
+tags:
+- base_model:adapter:unsloth/meta-llama-3.1-8b-instruct-bnb-4bit
+- lora
+- sft
+- transformers
+- trl
+- unsloth
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.18.1

experiments/de-en-pt-zh/adapter/adapter_config.json

@@ -0,0 +1,50 @@
+{
+  "alora_invocation_tokens": null,
+  "alpha_pattern": {},
+  "arrow_config": null,
+  "auto_mapping": {
+    "base_model_class": "LlamaForCausalLM",
+    "parent_library": "transformers.models.llama.modeling_llama",
+    "unsloth_fixed": true
+  },
+  "base_model_name_or_path": "unsloth/meta-llama-3.1-8b-instruct-bnb-4bit",
+  "bias": "none",
+  "corda_config": null,
+  "ensure_weight_tying": false,
+  "eva_config": null,
+  "exclude_modules": null,
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 16,
+  "lora_bias": false,
+  "lora_dropout": 0.1,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "peft_version": "0.18.1",
+  "qalora_group_size": 16,
+  "r": 8,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "gate_proj",
+    "v_proj",
+    "o_proj",
+    "k_proj",
+    "up_proj",
+    "down_proj",
+    "q_proj"
+  ],
+  "target_parameters": null,
+  "task_type": "CAUSAL_LM",
+  "trainable_token_indices": null,
+  "use_dora": false,
+  "use_qalora": false,
+  "use_rslora": true
+}

experiments/de-en-pt-zh/adapter/adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:051126fb9ec33a14e0ca6905264f0cdf6fa6a95969cce1b5cbcee93a06dd58c9
+size 83945296

experiments/de-en-pt-zh/adapter/chat_template.jinja

@@ -0,0 +1,109 @@
+{{- bos_token }}
+{%- if custom_tools is defined %}
+    {%- set tools = custom_tools %}
+{%- endif %}
+{%- if not tools_in_user_message is defined %}
+    {%- set tools_in_user_message = true %}
+{%- endif %}
+{%- if not date_string is defined %}
+    {%- set date_string = "26 Jul 2024" %}
+{%- endif %}
+{%- if not tools is defined %}
+    {%- set tools = none %}
+{%- endif %}
+
+{#- This block extracts the system message, so we can slot it into the right place. #}
+{%- if messages[0]['role'] == 'system' %}
+    {%- set system_message = messages[0]['content']|trim %}
+    {%- set messages = messages[1:] %}
+{%- else %}
+    {%- set system_message = "" %}
+{%- endif %}
+
+{#- System message + builtin tools #}
+{{- "<|start_header_id|>system<|end_header_id|>\n\n" }}
+{%- if builtin_tools is defined or tools is not none %}
+    {{- "Environment: ipython\n" }}
+{%- endif %}
+{%- if builtin_tools is defined %}
+    {{- "Tools: " + builtin_tools | reject('equalto', 'code_interpreter') | join(", ") + "\n\n"}}
+{%- endif %}
+{{- "Cutting Knowledge Date: December 2023\n" }}
+{{- "Today Date: " + date_string + "\n\n" }}
+{%- if tools is not none and not tools_in_user_message %}
+    {{- "You have access to the following functions. To call a function, please respond with JSON for a function call." }}
+    {{- 'Respond in the format {"name": function name, "parameters": dictionary of argument name and its value}.' }}
+    {{- "Do not use variables.\n\n" }}
+    {%- for t in tools %}
+        {{- t | tojson(indent=4) }}
+        {{- "\n\n" }}
+    {%- endfor %}
+{%- endif %}
+{{- system_message }}
+{{- "<|eot_id|>" }}
+
+{#- Custom tools are passed in a user message with some extra guidance #}
+{%- if tools_in_user_message and not tools is none %}
+    {#- Extract the first user message so we can plug it in here #}
+    {%- if messages | length != 0 %}
+        {%- set first_user_message = messages[0]['content']|trim %}
+        {%- set messages = messages[1:] %}
+    {%- else %}
+        {{- raise_exception("Cannot put tools in the first user message when there's no first user message!") }}
+{%- endif %}
+    {{- '<|start_header_id|>user<|end_header_id|>\n\n' -}}
+    {{- "Given the following functions, please respond with a JSON for a function call " }}
+    {{- "with its proper arguments that best answers the given prompt.\n\n" }}
+    {{- 'Respond in the format {"name": function name, "parameters": dictionary of argument name and its value}.' }}
+    {{- "Do not use variables.\n\n" }}
+    {%- for t in tools %}
+        {{- t | tojson(indent=4) }}
+        {{- "\n\n" }}
+    {%- endfor %}
+    {{- first_user_message + "<|eot_id|>"}}
+{%- endif %}
+
+{%- for message in messages %}
+    {%- if not (message.role == 'ipython' or message.role == 'tool' or 'tool_calls' in message) %}
+        {{- '<|start_header_id|>' + message['role'] + '<|end_header_id|>\n\n'+ message['content'] | trim + '<|eot_id|>' }}
+    {%- elif 'tool_calls' in message %}
+        {%- if not message.tool_calls|length == 1 %}
+            {{- raise_exception("This model only supports single tool-calls at once!") }}
+        {%- endif %}
+        {%- set tool_call = message.tool_calls[0].function %}
+        {%- if builtin_tools is defined and tool_call.name in builtin_tools %}
+            {{- '<|start_header_id|>assistant<|end_header_id|>\n\n' -}}
+            {{- "<|python_tag|>" + tool_call.name + ".call(" }}
+            {%- for arg_name, arg_val in tool_call.arguments | items %}
+                {{- arg_name + '="' + arg_val + '"' }}
+                {%- if not loop.last %}
+                    {{- ", " }}
+                {%- endif %}
+                {%- endfor %}
+            {{- ")" }}
+        {%- else  %}
+            {{- '<|start_header_id|>assistant<|end_header_id|>\n\n' -}}
+            {{- '{"name": "' + tool_call.name + '", ' }}
+            {{- '"parameters": ' }}
+            {{- tool_call.arguments | tojson }}
+            {{- "}" }}
+        {%- endif %}
+        {%- if builtin_tools is defined %}
+            {#- This means we're in ipython mode #}
+            {{- "<|eom_id|>" }}
+        {%- else %}
+            {{- "<|eot_id|>" }}
+        {%- endif %}
+    {%- elif message.role == "tool" or message.role == "ipython" %}
+        {{- "<|start_header_id|>ipython<|end_header_id|>\n\n" }}
+        {%- if message.content is mapping or message.content is iterable %}
+            {{- message.content | tojson }}
+        {%- else %}
+            {{- message.content }}
+        {%- endif %}
+        {{- "<|eot_id|>" }}
+    {%- endif %}
+{%- endfor %}
+{%- if add_generation_prompt %}
+    {{- '<|start_header_id|>assistant<|end_header_id|>\n\n' }}
+{%- endif %}

experiments/de-en-pt-zh/adapter/special_tokens_map.json

@@ -0,0 +1,23 @@
+{
+  "bos_token": {
+    "content": "<|begin_of_text|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "<|eot_id|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "<|finetune_right_pad_id|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

experiments/de-en-pt-zh/adapter/tokenizer.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6b9e4e7fb171f92fd137b777cc2714bf87d11576700a1dcd7a399e7bbe39537b
+size 17209920

experiments/de-en-pt-zh/adapter/tokenizer_config.json

@@ -0,0 +1,2066 @@
+{
+  "add_bos_token": true,
+  "added_tokens_decoder": {
+    "128000": {
+      "content": "<|begin_of_text|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128001": {
+      "content": "<|end_of_text|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128002": {
+      "content": "<|reserved_special_token_0|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128003": {
+      "content": "<|reserved_special_token_1|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128004": {
+      "content": "<|finetune_right_pad_id|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128005": {
+      "content": "<|reserved_special_token_2|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128006": {
+      "content": "<|start_header_id|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128007": {
+      "content": "<|end_header_id|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128008": {
+      "content": "<|eom_id|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128009": {
+      "content": "<|eot_id|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128010": {
+      "content": "<|python_tag|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128011": {
+      "content": "<|reserved_special_token_3|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128012": {
+      "content": "<|reserved_special_token_4|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128013": {
+      "content": "<|reserved_special_token_5|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128014": {
+      "content": "<|reserved_special_token_6|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128015": {
+      "content": "<|reserved_special_token_7|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128016": {
+      "content": "<|reserved_special_token_8|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128017": {
+      "content": "<|reserved_special_token_9|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128018": {
+      "content": "<|reserved_special_token_10|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128019": {
+      "content": "<|reserved_special_token_11|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128020": {
+      "content": "<|reserved_special_token_12|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128021": {
+      "content": "<|reserved_special_token_13|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128022": {
+      "content": "<|reserved_special_token_14|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128023": {
+      "content": "<|reserved_special_token_15|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128024": {
+      "content": "<|reserved_special_token_16|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128025": {
+      "content": "<|reserved_special_token_17|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128026": {
+      "content": "<|reserved_special_token_18|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128027": {
+      "content": "<|reserved_special_token_19|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128028": {
+      "content": "<|reserved_special_token_20|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128029": {
+      "content": "<|reserved_special_token_21|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128030": {
+      "content": "<|reserved_special_token_22|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128031": {
+      "content": "<|reserved_special_token_23|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128032": {
+      "content": "<|reserved_special_token_24|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128033": {
+      "content": "<|reserved_special_token_25|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128034": {
+      "content": "<|reserved_special_token_26|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128035": {
+      "content": "<|reserved_special_token_27|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128036": {
+      "content": "<|reserved_special_token_28|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128037": {
+      "content": "<|reserved_special_token_29|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128038": {
+      "content": "<|reserved_special_token_30|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128039": {
+      "content": "<|reserved_special_token_31|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128040": {
+      "content": "<|reserved_special_token_32|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128041": {
+      "content": "<|reserved_special_token_33|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128042": {
+      "content": "<|reserved_special_token_34|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128043": {
+      "content": "<|reserved_special_token_35|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128044": {
+      "content": "<|reserved_special_token_36|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128045": {
+      "content": "<|reserved_special_token_37|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128046": {
+      "content": "<|reserved_special_token_38|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128047": {
+      "content": "<|reserved_special_token_39|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128048": {
+      "content": "<|reserved_special_token_40|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128049": {
+      "content": "<|reserved_special_token_41|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128050": {
+      "content": "<|reserved_special_token_42|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128051": {
+      "content": "<|reserved_special_token_43|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128052": {
+      "content": "<|reserved_special_token_44|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128053": {
+      "content": "<|reserved_special_token_45|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128054": {
+      "content": "<|reserved_special_token_46|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128055": {
+      "content": "<|reserved_special_token_47|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128056": {
+      "content": "<|reserved_special_token_48|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128057": {
+      "content": "<|reserved_special_token_49|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128058": {
+      "content": "<|reserved_special_token_50|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128059": {
+      "content": "<|reserved_special_token_51|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128060": {
+      "content": "<|reserved_special_token_52|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128061": {
+      "content": "<|reserved_special_token_53|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128062": {
+      "content": "<|reserved_special_token_54|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128063": {
+      "content": "<|reserved_special_token_55|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128064": {
+      "content": "<|reserved_special_token_56|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128065": {
+      "content": "<|reserved_special_token_57|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128066": {
+      "content": "<|reserved_special_token_58|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128067": {
+      "content": "<|reserved_special_token_59|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128068": {
+      "content": "<|reserved_special_token_60|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128069": {
+      "content": "<|reserved_special_token_61|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128070": {
+      "content": "<|reserved_special_token_62|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128071": {
+      "content": "<|reserved_special_token_63|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128072": {
+      "content": "<|reserved_special_token_64|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128073": {
+      "content": "<|reserved_special_token_65|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128074": {
+      "content": "<|reserved_special_token_66|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128075": {
+      "content": "<|reserved_special_token_67|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128076": {
+      "content": "<|reserved_special_token_68|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128077": {
+      "content": "<|reserved_special_token_69|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128078": {
+      "content": "<|reserved_special_token_70|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128079": {
+      "content": "<|reserved_special_token_71|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128080": {
+      "content": "<|reserved_special_token_72|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128081": {
+      "content": "<|reserved_special_token_73|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128082": {
+      "content": "<|reserved_special_token_74|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128083": {
+      "content": "<|reserved_special_token_75|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128084": {
+      "content": "<|reserved_special_token_76|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128085": {
+      "content": "<|reserved_special_token_77|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128086": {
+      "content": "<|reserved_special_token_78|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128087": {
+      "content": "<|reserved_special_token_79|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128088": {
+      "content": "<|reserved_special_token_80|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128089": {
+      "content": "<|reserved_special_token_81|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128090": {
+      "content": "<|reserved_special_token_82|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128091": {
+      "content": "<|reserved_special_token_83|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128092": {
+      "content": "<|reserved_special_token_84|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128093": {
+      "content": "<|reserved_special_token_85|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128094": {
+      "content": "<|reserved_special_token_86|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128095": {
+      "content": "<|reserved_special_token_87|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128096": {
+      "content": "<|reserved_special_token_88|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128097": {
+      "content": "<|reserved_special_token_89|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128098": {
+      "content": "<|reserved_special_token_90|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128099": {
+      "content": "<|reserved_special_token_91|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128100": {
+      "content": "<|reserved_special_token_92|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128101": {
+      "content": "<|reserved_special_token_93|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128102": {
+      "content": "<|reserved_special_token_94|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128103": {
+      "content": "<|reserved_special_token_95|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128104": {
+      "content": "<|reserved_special_token_96|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128105": {
+      "content": "<|reserved_special_token_97|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128106": {
+      "content": "<|reserved_special_token_98|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128107": {
+      "content": "<|reserved_special_token_99|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128108": {
+      "content": "<|reserved_special_token_100|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128109": {
+      "content": "<|reserved_special_token_101|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128110": {
+      "content": "<|reserved_special_token_102|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128111": {
+      "content": "<|reserved_special_token_103|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128112": {
+      "content": "<|reserved_special_token_104|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128113": {
+      "content": "<|reserved_special_token_105|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128114": {
+      "content": "<|reserved_special_token_106|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128115": {
+      "content": "<|reserved_special_token_107|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128116": {
+      "content": "<|reserved_special_token_108|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128117": {
+      "content": "<|reserved_special_token_109|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128118": {
+      "content": "<|reserved_special_token_110|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128119": {
+      "content": "<|reserved_special_token_111|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128120": {
+      "content": "<|reserved_special_token_112|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128121": {
+      "content": "<|reserved_special_token_113|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128122": {
+      "content": "<|reserved_special_token_114|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128123": {
+      "content": "<|reserved_special_token_115|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128124": {
+      "content": "<|reserved_special_token_116|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128125": {
+      "content": "<|reserved_special_token_117|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128126": {
+      "content": "<|reserved_special_token_118|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128127": {
+      "content": "<|reserved_special_token_119|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128128": {
+      "content": "<|reserved_special_token_120|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128129": {
+      "content": "<|reserved_special_token_121|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128130": {
+      "content": "<|reserved_special_token_122|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128131": {
+      "content": "<|reserved_special_token_123|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128132": {
+      "content": "<|reserved_special_token_124|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128133": {
+      "content": "<|reserved_special_token_125|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128134": {
+      "content": "<|reserved_special_token_126|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128135": {
+      "content": "<|reserved_special_token_127|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128136": {
+      "content": "<|reserved_special_token_128|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128137": {
+      "content": "<|reserved_special_token_129|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128138": {
+      "content": "<|reserved_special_token_130|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128139": {
+      "content": "<|reserved_special_token_131|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128140": {
+      "content": "<|reserved_special_token_132|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128141": {
+      "content": "<|reserved_special_token_133|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128142": {
+      "content": "<|reserved_special_token_134|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128143": {
+      "content": "<|reserved_special_token_135|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128144": {
+      "content": "<|reserved_special_token_136|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128145": {
+      "content": "<|reserved_special_token_137|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128146": {
+      "content": "<|reserved_special_token_138|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128147": {
+      "content": "<|reserved_special_token_139|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128148": {
+      "content": "<|reserved_special_token_140|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128149": {
+      "content": "<|reserved_special_token_141|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128150": {
+      "content": "<|reserved_special_token_142|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128151": {
+      "content": "<|reserved_special_token_143|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128152": {
+      "content": "<|reserved_special_token_144|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128153": {
+      "content": "<|reserved_special_token_145|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128154": {
+      "content": "<|reserved_special_token_146|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128155": {
+      "content": "<|reserved_special_token_147|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128156": {
+      "content": "<|reserved_special_token_148|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128157": {
+      "content": "<|reserved_special_token_149|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128158": {
+      "content": "<|reserved_special_token_150|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128159": {
+      "content": "<|reserved_special_token_151|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128160": {
+      "content": "<|reserved_special_token_152|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128161": {
+      "content": "<|reserved_special_token_153|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128162": {
+      "content": "<|reserved_special_token_154|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128163": {
+      "content": "<|reserved_special_token_155|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128164": {
+      "content": "<|reserved_special_token_156|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128165": {
+      "content": "<|reserved_special_token_157|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128166": {
+      "content": "<|reserved_special_token_158|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128167": {
+      "content": "<|reserved_special_token_159|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128168": {
+      "content": "<|reserved_special_token_160|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128169": {
+      "content": "<|reserved_special_token_161|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128170": {
+      "content": "<|reserved_special_token_162|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128171": {
+      "content": "<|reserved_special_token_163|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128172": {
+      "content": "<|reserved_special_token_164|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128173": {
+      "content": "<|reserved_special_token_165|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128174": {
+      "content": "<|reserved_special_token_166|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128175": {
+      "content": "<|reserved_special_token_167|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128176": {
+      "content": "<|reserved_special_token_168|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128177": {
+      "content": "<|reserved_special_token_169|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128178": {
+      "content": "<|reserved_special_token_170|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128179": {
+      "content": "<|reserved_special_token_171|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128180": {
+      "content": "<|reserved_special_token_172|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128181": {
+      "content": "<|reserved_special_token_173|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128182": {
+      "content": "<|reserved_special_token_174|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128183": {
+      "content": "<|reserved_special_token_175|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128184": {
+      "content": "<|reserved_special_token_176|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128185": {
+      "content": "<|reserved_special_token_177|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128186": {
+      "content": "<|reserved_special_token_178|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128187": {
+      "content": "<|reserved_special_token_179|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128188": {
+      "content": "<|reserved_special_token_180|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128189": {
+      "content": "<|reserved_special_token_181|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128190": {
+      "content": "<|reserved_special_token_182|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128191": {
+      "content": "<|reserved_special_token_183|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128192": {
+      "content": "<|reserved_special_token_184|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128193": {
+      "content": "<|reserved_special_token_185|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128194": {
+      "content": "<|reserved_special_token_186|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128195": {
+      "content": "<|reserved_special_token_187|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128196": {
+      "content": "<|reserved_special_token_188|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128197": {
+      "content": "<|reserved_special_token_189|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128198": {
+      "content": "<|reserved_special_token_190|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128199": {
+      "content": "<|reserved_special_token_191|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128200": {
+      "content": "<|reserved_special_token_192|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128201": {
+      "content": "<|reserved_special_token_193|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128202": {
+      "content": "<|reserved_special_token_194|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128203": {
+      "content": "<|reserved_special_token_195|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128204": {
+      "content": "<|reserved_special_token_196|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128205": {
+      "content": "<|reserved_special_token_197|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128206": {
+      "content": "<|reserved_special_token_198|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128207": {
+      "content": "<|reserved_special_token_199|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128208": {
+      "content": "<|reserved_special_token_200|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128209": {
+      "content": "<|reserved_special_token_201|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128210": {
+      "content": "<|reserved_special_token_202|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128211": {
+      "content": "<|reserved_special_token_203|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128212": {
+      "content": "<|reserved_special_token_204|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128213": {
+      "content": "<|reserved_special_token_205|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128214": {
+      "content": "<|reserved_special_token_206|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128215": {
+      "content": "<|reserved_special_token_207|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128216": {
+      "content": "<|reserved_special_token_208|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128217": {
+      "content": "<|reserved_special_token_209|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128218": {
+      "content": "<|reserved_special_token_210|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128219": {
+      "content": "<|reserved_special_token_211|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128220": {
+      "content": "<|reserved_special_token_212|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128221": {
+      "content": "<|reserved_special_token_213|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128222": {
+      "content": "<|reserved_special_token_214|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128223": {
+      "content": "<|reserved_special_token_215|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128224": {
+      "content": "<|reserved_special_token_216|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128225": {
+      "content": "<|reserved_special_token_217|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128226": {
+      "content": "<|reserved_special_token_218|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128227": {
+      "content": "<|reserved_special_token_219|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128228": {
+      "content": "<|reserved_special_token_220|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128229": {
+      "content": "<|reserved_special_token_221|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128230": {
+      "content": "<|reserved_special_token_222|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128231": {
+      "content": "<|reserved_special_token_223|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128232": {
+      "content": "<|reserved_special_token_224|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128233": {
+      "content": "<|reserved_special_token_225|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128234": {
+      "content": "<|reserved_special_token_226|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128235": {
+      "content": "<|reserved_special_token_227|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128236": {
+      "content": "<|reserved_special_token_228|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128237": {
+      "content": "<|reserved_special_token_229|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128238": {
+      "content": "<|reserved_special_token_230|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128239": {
+      "content": "<|reserved_special_token_231|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128240": {
+      "content": "<|reserved_special_token_232|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128241": {
+      "content": "<|reserved_special_token_233|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128242": {
+      "content": "<|reserved_special_token_234|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128243": {
+      "content": "<|reserved_special_token_235|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128244": {
+      "content": "<|reserved_special_token_236|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128245": {
+      "content": "<|reserved_special_token_237|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128246": {
+      "content": "<|reserved_special_token_238|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128247": {
+      "content": "<|reserved_special_token_239|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128248": {
+      "content": "<|reserved_special_token_240|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128249": {
+      "content": "<|reserved_special_token_241|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128250": {
+      "content": "<|reserved_special_token_242|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128251": {
+      "content": "<|reserved_special_token_243|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128252": {
+      "content": "<|reserved_special_token_244|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128253": {
+      "content": "<|reserved_special_token_245|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128254": {
+      "content": "<|reserved_special_token_246|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128255": {
+      "content": "<|reserved_special_token_247|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "bos_token": "<|begin_of_text|>",
+  "clean_up_tokenization_spaces": true,
+  "eos_token": "<|eot_id|>",
+  "extra_special_tokens": {},
+  "model_input_names": [
+    "input_ids",
+    "attention_mask"
+  ],
+  "model_max_length": 131072,
+  "pad_token": "<|finetune_right_pad_id|>",
+  "padding_side": "left",
+  "tokenizer_class": "PreTrainedTokenizerFast",
+  "unk_token": null
+}

experiments/de-en-pt-zh/checkpoint-1000/README.md

@@ -0,0 +1,210 @@
+---
+base_model: unsloth/meta-llama-3.1-8b-instruct-bnb-4bit
+library_name: peft
+pipeline_tag: text-generation
+tags:
+- base_model:adapter:unsloth/meta-llama-3.1-8b-instruct-bnb-4bit
+- lora
+- sft
+- transformers
+- trl
+- unsloth
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.18.1

experiments/de-en-pt-zh/checkpoint-1000/adapter_config.json

@@ -0,0 +1,50 @@
+{
+  "alora_invocation_tokens": null,
+  "alpha_pattern": {},
+  "arrow_config": null,
+  "auto_mapping": {
+    "base_model_class": "LlamaForCausalLM",
+    "parent_library": "transformers.models.llama.modeling_llama",
+    "unsloth_fixed": true
+  },
+  "base_model_name_or_path": "unsloth/meta-llama-3.1-8b-instruct-bnb-4bit",
+  "bias": "none",
+  "corda_config": null,
+  "ensure_weight_tying": false,
+  "eva_config": null,
+  "exclude_modules": null,
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 16,
+  "lora_bias": false,
+  "lora_dropout": 0.1,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "peft_version": "0.18.1",
+  "qalora_group_size": 16,
+  "r": 8,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "gate_proj",
+    "v_proj",
+    "o_proj",
+    "k_proj",
+    "up_proj",
+    "down_proj",
+    "q_proj"
+  ],
+  "target_parameters": null,
+  "task_type": "CAUSAL_LM",
+  "trainable_token_indices": null,
+  "use_dora": false,
+  "use_qalora": false,
+  "use_rslora": true
+}

experiments/de-en-pt-zh/checkpoint-1000/adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:834fddf3f3b4f812d71ea789ec5f3bbd9bcfa2780a0b2b89304b836e24b58e0c
+size 83945296

experiments/de-en-pt-zh/checkpoint-1000/chat_template.jinja

@@ -0,0 +1,109 @@
+{{- bos_token }}
+{%- if custom_tools is defined %}
+    {%- set tools = custom_tools %}
+{%- endif %}
+{%- if not tools_in_user_message is defined %}
+    {%- set tools_in_user_message = true %}
+{%- endif %}
+{%- if not date_string is defined %}
+    {%- set date_string = "26 Jul 2024" %}
+{%- endif %}
+{%- if not tools is defined %}
+    {%- set tools = none %}
+{%- endif %}
+
+{#- This block extracts the system message, so we can slot it into the right place. #}
+{%- if messages[0]['role'] == 'system' %}
+    {%- set system_message = messages[0]['content']|trim %}
+    {%- set messages = messages[1:] %}
+{%- else %}
+    {%- set system_message = "" %}
+{%- endif %}
+
+{#- System message + builtin tools #}
+{{- "<|start_header_id|>system<|end_header_id|>\n\n" }}
+{%- if builtin_tools is defined or tools is not none %}
+    {{- "Environment: ipython\n" }}
+{%- endif %}
+{%- if builtin_tools is defined %}
+    {{- "Tools: " + builtin_tools | reject('equalto', 'code_interpreter') | join(", ") + "\n\n"}}
+{%- endif %}
+{{- "Cutting Knowledge Date: December 2023\n" }}
+{{- "Today Date: " + date_string + "\n\n" }}
+{%- if tools is not none and not tools_in_user_message %}
+    {{- "You have access to the following functions. To call a function, please respond with JSON for a function call." }}
+    {{- 'Respond in the format {"name": function name, "parameters": dictionary of argument name and its value}.' }}
+    {{- "Do not use variables.\n\n" }}
+    {%- for t in tools %}
+        {{- t | tojson(indent=4) }}
+        {{- "\n\n" }}
+    {%- endfor %}
+{%- endif %}
+{{- system_message }}
+{{- "<|eot_id|>" }}
+
+{#- Custom tools are passed in a user message with some extra guidance #}
+{%- if tools_in_user_message and not tools is none %}
+    {#- Extract the first user message so we can plug it in here #}
+    {%- if messages | length != 0 %}
+        {%- set first_user_message = messages[0]['content']|trim %}
+        {%- set messages = messages[1:] %}
+    {%- else %}
+        {{- raise_exception("Cannot put tools in the first user message when there's no first user message!") }}
+{%- endif %}
+    {{- '<|start_header_id|>user<|end_header_id|>\n\n' -}}
+    {{- "Given the following functions, please respond with a JSON for a function call " }}
+    {{- "with its proper arguments that best answers the given prompt.\n\n" }}
+    {{- 'Respond in the format {"name": function name, "parameters": dictionary of argument name and its value}.' }}
+    {{- "Do not use variables.\n\n" }}
+    {%- for t in tools %}
+        {{- t | tojson(indent=4) }}
+        {{- "\n\n" }}
+    {%- endfor %}
+    {{- first_user_message + "<|eot_id|>"}}
+{%- endif %}
+
+{%- for message in messages %}
+    {%- if not (message.role == 'ipython' or message.role == 'tool' or 'tool_calls' in message) %}
+        {{- '<|start_header_id|>' + message['role'] + '<|end_header_id|>\n\n'+ message['content'] | trim + '<|eot_id|>' }}
+    {%- elif 'tool_calls' in message %}
+        {%- if not message.tool_calls|length == 1 %}
+            {{- raise_exception("This model only supports single tool-calls at once!") }}
+        {%- endif %}
+        {%- set tool_call = message.tool_calls[0].function %}
+        {%- if builtin_tools is defined and tool_call.name in builtin_tools %}
+            {{- '<|start_header_id|>assistant<|end_header_id|>\n\n' -}}
+            {{- "<|python_tag|>" + tool_call.name + ".call(" }}
+            {%- for arg_name, arg_val in tool_call.arguments | items %}
+                {{- arg_name + '="' + arg_val + '"' }}
+                {%- if not loop.last %}
+                    {{- ", " }}
+                {%- endif %}
+                {%- endfor %}
+            {{- ")" }}
+        {%- else  %}
+            {{- '<|start_header_id|>assistant<|end_header_id|>\n\n' -}}
+            {{- '{"name": "' + tool_call.name + '", ' }}
+            {{- '"parameters": ' }}
+            {{- tool_call.arguments | tojson }}
+            {{- "}" }}
+        {%- endif %}
+        {%- if builtin_tools is defined %}
+            {#- This means we're in ipython mode #}
+            {{- "<|eom_id|>" }}
+        {%- else %}
+            {{- "<|eot_id|>" }}
+        {%- endif %}
+    {%- elif message.role == "tool" or message.role == "ipython" %}
+        {{- "<|start_header_id|>ipython<|end_header_id|>\n\n" }}
+        {%- if message.content is mapping or message.content is iterable %}
+            {{- message.content | tojson }}
+        {%- else %}
+            {{- message.content }}
+        {%- endif %}
+        {{- "<|eot_id|>" }}
+    {%- endif %}
+{%- endfor %}
+{%- if add_generation_prompt %}
+    {{- '<|start_header_id|>assistant<|end_header_id|>\n\n' }}
+{%- endif %}

experiments/de-en-pt-zh/checkpoint-1000/optimizer.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e22cd44cbd6079206a3bdf28cd9def7b2fcd700d31537fcffc8b2cc7c716eff4
+size 43123429

experiments/de-en-pt-zh/checkpoint-1000/rng_state.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ead85b09410a86a07e70ed121bc4123b9656db163ec170d8c68860f53d32663b
+size 14645

experiments/de-en-pt-zh/checkpoint-1000/scheduler.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f4b83d2b61763737893f22a960e5241e4b9d5d7cf00b6381e3c81c0dd6945f88
+size 1465

experiments/de-en-pt-zh/checkpoint-1000/special_tokens_map.json

@@ -0,0 +1,23 @@
+{
+  "bos_token": {
+    "content": "<|begin_of_text|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "<|eot_id|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "<|finetune_right_pad_id|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

experiments/de-en-pt-zh/checkpoint-1000/tokenizer.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6b9e4e7fb171f92fd137b777cc2714bf87d11576700a1dcd7a399e7bbe39537b
+size 17209920

experiments/de-en-pt-zh/checkpoint-1000/tokenizer_config.json

@@ -0,0 +1,2066 @@
+{
+  "add_bos_token": true,
+  "added_tokens_decoder": {
+    "128000": {
+      "content": "<|begin_of_text|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128001": {
+      "content": "<|end_of_text|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128002": {
+      "content": "<|reserved_special_token_0|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128003": {
+      "content": "<|reserved_special_token_1|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128004": {
+      "content": "<|finetune_right_pad_id|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128005": {
+      "content": "<|reserved_special_token_2|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128006": {
+      "content": "<|start_header_id|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128007": {
+      "content": "<|end_header_id|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128008": {
+      "content": "<|eom_id|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128009": {
+      "content": "<|eot_id|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128010": {
+      "content": "<|python_tag|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128011": {
+      "content": "<|reserved_special_token_3|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128012": {
+      "content": "<|reserved_special_token_4|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128013": {
+      "content": "<|reserved_special_token_5|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128014": {
+      "content": "<|reserved_special_token_6|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128015": {
+      "content": "<|reserved_special_token_7|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128016": {
+      "content": "<|reserved_special_token_8|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128017": {
+      "content": "<|reserved_special_token_9|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128018": {
+      "content": "<|reserved_special_token_10|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128019": {
+      "content": "<|reserved_special_token_11|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128020": {
+      "content": "<|reserved_special_token_12|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128021": {
+      "content": "<|reserved_special_token_13|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128022": {
+      "content": "<|reserved_special_token_14|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128023": {
+      "content": "<|reserved_special_token_15|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128024": {
+      "content": "<|reserved_special_token_16|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128025": {
+      "content": "<|reserved_special_token_17|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128026": {
+      "content": "<|reserved_special_token_18|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128027": {
+      "content": "<|reserved_special_token_19|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128028": {
+      "content": "<|reserved_special_token_20|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128029": {
+      "content": "<|reserved_special_token_21|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128030": {
+      "content": "<|reserved_special_token_22|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128031": {
+      "content": "<|reserved_special_token_23|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128032": {
+      "content": "<|reserved_special_token_24|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128033": {
+      "content": "<|reserved_special_token_25|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128034": {
+      "content": "<|reserved_special_token_26|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128035": {
+      "content": "<|reserved_special_token_27|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128036": {
+      "content": "<|reserved_special_token_28|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128037": {
+      "content": "<|reserved_special_token_29|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128038": {
+      "content": "<|reserved_special_token_30|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128039": {
+      "content": "<|reserved_special_token_31|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128040": {
+      "content": "<|reserved_special_token_32|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128041": {
+      "content": "<|reserved_special_token_33|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128042": {
+      "content": "<|reserved_special_token_34|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128043": {
+      "content": "<|reserved_special_token_35|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128044": {
+      "content": "<|reserved_special_token_36|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128045": {
+      "content": "<|reserved_special_token_37|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128046": {
+      "content": "<|reserved_special_token_38|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128047": {
+      "content": "<|reserved_special_token_39|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128048": {
+      "content": "<|reserved_special_token_40|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128049": {
+      "content": "<|reserved_special_token_41|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128050": {
+      "content": "<|reserved_special_token_42|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128051": {
+      "content": "<|reserved_special_token_43|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128052": {
+      "content": "<|reserved_special_token_44|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128053": {
+      "content": "<|reserved_special_token_45|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128054": {
+      "content": "<|reserved_special_token_46|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128055": {
+      "content": "<|reserved_special_token_47|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128056": {
+      "content": "<|reserved_special_token_48|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128057": {
+      "content": "<|reserved_special_token_49|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128058": {
+      "content": "<|reserved_special_token_50|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128059": {
+      "content": "<|reserved_special_token_51|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128060": {
+      "content": "<|reserved_special_token_52|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128061": {
+      "content": "<|reserved_special_token_53|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128062": {
+      "content": "<|reserved_special_token_54|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128063": {
+      "content": "<|reserved_special_token_55|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128064": {
+      "content": "<|reserved_special_token_56|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128065": {
+      "content": "<|reserved_special_token_57|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128066": {
+      "content": "<|reserved_special_token_58|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128067": {
+      "content": "<|reserved_special_token_59|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128068": {
+      "content": "<|reserved_special_token_60|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128069": {
+      "content": "<|reserved_special_token_61|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128070": {
+      "content": "<|reserved_special_token_62|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128071": {
+      "content": "<|reserved_special_token_63|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128072": {
+      "content": "<|reserved_special_token_64|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128073": {
+      "content": "<|reserved_special_token_65|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128074": {
+      "content": "<|reserved_special_token_66|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128075": {
+      "content": "<|reserved_special_token_67|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128076": {
+      "content": "<|reserved_special_token_68|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128077": {
+      "content": "<|reserved_special_token_69|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128078": {
+      "content": "<|reserved_special_token_70|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128079": {
+      "content": "<|reserved_special_token_71|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128080": {
+      "content": "<|reserved_special_token_72|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128081": {
+      "content": "<|reserved_special_token_73|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128082": {
+      "content": "<|reserved_special_token_74|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128083": {
+      "content": "<|reserved_special_token_75|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128084": {
+      "content": "<|reserved_special_token_76|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128085": {
+      "content": "<|reserved_special_token_77|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128086": {
+      "content": "<|reserved_special_token_78|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128087": {
+      "content": "<|reserved_special_token_79|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128088": {
+      "content": "<|reserved_special_token_80|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128089": {
+      "content": "<|reserved_special_token_81|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128090": {
+      "content": "<|reserved_special_token_82|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128091": {
+      "content": "<|reserved_special_token_83|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128092": {
+      "content": "<|reserved_special_token_84|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128093": {
+      "content": "<|reserved_special_token_85|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128094": {
+      "content": "<|reserved_special_token_86|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128095": {
+      "content": "<|reserved_special_token_87|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128096": {
+      "content": "<|reserved_special_token_88|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128097": {
+      "content": "<|reserved_special_token_89|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128098": {
+      "content": "<|reserved_special_token_90|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128099": {
+      "content": "<|reserved_special_token_91|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128100": {
+      "content": "<|reserved_special_token_92|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128101": {
+      "content": "<|reserved_special_token_93|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128102": {
+      "content": "<|reserved_special_token_94|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128103": {
+      "content": "<|reserved_special_token_95|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128104": {
+      "content": "<|reserved_special_token_96|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128105": {
+      "content": "<|reserved_special_token_97|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128106": {
+      "content": "<|reserved_special_token_98|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128107": {
+      "content": "<|reserved_special_token_99|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128108": {
+      "content": "<|reserved_special_token_100|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128109": {
+      "content": "<|reserved_special_token_101|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128110": {
+      "content": "<|reserved_special_token_102|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128111": {
+      "content": "<|reserved_special_token_103|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128112": {
+      "content": "<|reserved_special_token_104|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128113": {
+      "content": "<|reserved_special_token_105|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128114": {
+      "content": "<|reserved_special_token_106|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128115": {
+      "content": "<|reserved_special_token_107|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128116": {
+      "content": "<|reserved_special_token_108|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128117": {
+      "content": "<|reserved_special_token_109|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128118": {
+      "content": "<|reserved_special_token_110|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128119": {
+      "content": "<|reserved_special_token_111|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128120": {
+      "content": "<|reserved_special_token_112|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128121": {
+      "content": "<|reserved_special_token_113|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128122": {
+      "content": "<|reserved_special_token_114|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128123": {
+      "content": "<|reserved_special_token_115|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128124": {
+      "content": "<|reserved_special_token_116|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128125": {
+      "content": "<|reserved_special_token_117|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128126": {
+      "content": "<|reserved_special_token_118|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128127": {
+      "content": "<|reserved_special_token_119|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128128": {
+      "content": "<|reserved_special_token_120|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128129": {
+      "content": "<|reserved_special_token_121|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128130": {
+      "content": "<|reserved_special_token_122|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128131": {
+      "content": "<|reserved_special_token_123|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128132": {
+      "content": "<|reserved_special_token_124|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128133": {
+      "content": "<|reserved_special_token_125|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128134": {
+      "content": "<|reserved_special_token_126|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128135": {
+      "content": "<|reserved_special_token_127|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128136": {
+      "content": "<|reserved_special_token_128|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128137": {
+      "content": "<|reserved_special_token_129|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128138": {
+      "content": "<|reserved_special_token_130|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128139": {
+      "content": "<|reserved_special_token_131|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128140": {
+      "content": "<|reserved_special_token_132|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128141": {
+      "content": "<|reserved_special_token_133|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128142": {
+      "content": "<|reserved_special_token_134|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128143": {
+      "content": "<|reserved_special_token_135|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128144": {
+      "content": "<|reserved_special_token_136|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128145": {
+      "content": "<|reserved_special_token_137|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128146": {
+      "content": "<|reserved_special_token_138|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128147": {
+      "content": "<|reserved_special_token_139|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128148": {
+      "content": "<|reserved_special_token_140|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128149": {
+      "content": "<|reserved_special_token_141|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128150": {
+      "content": "<|reserved_special_token_142|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128151": {
+      "content": "<|reserved_special_token_143|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128152": {
+      "content": "<|reserved_special_token_144|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128153": {
+      "content": "<|reserved_special_token_145|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128154": {
+      "content": "<|reserved_special_token_146|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128155": {
+      "content": "<|reserved_special_token_147|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128156": {
+      "content": "<|reserved_special_token_148|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128157": {
+      "content": "<|reserved_special_token_149|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128158": {
+      "content": "<|reserved_special_token_150|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128159": {
+      "content": "<|reserved_special_token_151|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128160": {
+      "content": "<|reserved_special_token_152|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128161": {
+      "content": "<|reserved_special_token_153|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128162": {
+      "content": "<|reserved_special_token_154|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128163": {
+      "content": "<|reserved_special_token_155|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128164": {
+      "content": "<|reserved_special_token_156|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128165": {
+      "content": "<|reserved_special_token_157|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128166": {
+      "content": "<|reserved_special_token_158|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128167": {
+      "content": "<|reserved_special_token_159|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128168": {
+      "content": "<|reserved_special_token_160|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128169": {
+      "content": "<|reserved_special_token_161|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128170": {
+      "content": "<|reserved_special_token_162|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128171": {
+      "content": "<|reserved_special_token_163|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128172": {
+      "content": "<|reserved_special_token_164|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128173": {
+      "content": "<|reserved_special_token_165|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128174": {
+      "content": "<|reserved_special_token_166|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128175": {
+      "content": "<|reserved_special_token_167|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128176": {
+      "content": "<|reserved_special_token_168|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128177": {
+      "content": "<|reserved_special_token_169|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128178": {
+      "content": "<|reserved_special_token_170|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128179": {
+      "content": "<|reserved_special_token_171|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128180": {
+      "content": "<|reserved_special_token_172|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128181": {
+      "content": "<|reserved_special_token_173|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128182": {
+      "content": "<|reserved_special_token_174|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128183": {
+      "content": "<|reserved_special_token_175|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128184": {
+      "content": "<|reserved_special_token_176|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128185": {
+      "content": "<|reserved_special_token_177|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128186": {
+      "content": "<|reserved_special_token_178|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128187": {
+      "content": "<|reserved_special_token_179|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128188": {
+      "content": "<|reserved_special_token_180|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128189": {
+      "content": "<|reserved_special_token_181|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128190": {
+      "content": "<|reserved_special_token_182|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128191": {
+      "content": "<|reserved_special_token_183|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128192": {
+      "content": "<|reserved_special_token_184|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128193": {
+      "content": "<|reserved_special_token_185|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128194": {
+      "content": "<|reserved_special_token_186|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128195": {
+      "content": "<|reserved_special_token_187|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128196": {
+      "content": "<|reserved_special_token_188|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128197": {
+      "content": "<|reserved_special_token_189|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128198": {
+      "content": "<|reserved_special_token_190|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128199": {
+      "content": "<|reserved_special_token_191|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128200": {
+      "content": "<|reserved_special_token_192|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128201": {
+      "content": "<|reserved_special_token_193|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128202": {
+      "content": "<|reserved_special_token_194|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128203": {
+      "content": "<|reserved_special_token_195|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128204": {
+      "content": "<|reserved_special_token_196|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128205": {
+      "content": "<|reserved_special_token_197|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128206": {
+      "content": "<|reserved_special_token_198|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128207": {
+      "content": "<|reserved_special_token_199|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128208": {
+      "content": "<|reserved_special_token_200|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128209": {
+      "content": "<|reserved_special_token_201|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128210": {
+      "content": "<|reserved_special_token_202|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128211": {
+      "content": "<|reserved_special_token_203|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128212": {
+      "content": "<|reserved_special_token_204|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128213": {
+      "content": "<|reserved_special_token_205|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128214": {
+      "content": "<|reserved_special_token_206|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128215": {
+      "content": "<|reserved_special_token_207|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128216": {
+      "content": "<|reserved_special_token_208|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128217": {
+      "content": "<|reserved_special_token_209|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128218": {
+      "content": "<|reserved_special_token_210|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128219": {
+      "content": "<|reserved_special_token_211|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128220": {
+      "content": "<|reserved_special_token_212|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128221": {
+      "content": "<|reserved_special_token_213|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128222": {
+      "content": "<|reserved_special_token_214|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128223": {
+      "content": "<|reserved_special_token_215|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128224": {
+      "content": "<|reserved_special_token_216|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128225": {
+      "content": "<|reserved_special_token_217|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128226": {
+      "content": "<|reserved_special_token_218|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128227": {
+      "content": "<|reserved_special_token_219|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128228": {
+      "content": "<|reserved_special_token_220|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128229": {
+      "content": "<|reserved_special_token_221|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128230": {
+      "content": "<|reserved_special_token_222|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128231": {
+      "content": "<|reserved_special_token_223|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128232": {
+      "content": "<|reserved_special_token_224|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128233": {
+      "content": "<|reserved_special_token_225|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128234": {
+      "content": "<|reserved_special_token_226|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128235": {
+      "content": "<|reserved_special_token_227|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128236": {
+      "content": "<|reserved_special_token_228|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128237": {
+      "content": "<|reserved_special_token_229|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128238": {
+      "content": "<|reserved_special_token_230|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128239": {
+      "content": "<|reserved_special_token_231|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128240": {
+      "content": "<|reserved_special_token_232|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128241": {
+      "content": "<|reserved_special_token_233|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128242": {
+      "content": "<|reserved_special_token_234|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128243": {
+      "content": "<|reserved_special_token_235|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128244": {
+      "content": "<|reserved_special_token_236|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128245": {
+      "content": "<|reserved_special_token_237|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128246": {
+      "content": "<|reserved_special_token_238|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128247": {
+      "content": "<|reserved_special_token_239|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128248": {
+      "content": "<|reserved_special_token_240|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128249": {
+      "content": "<|reserved_special_token_241|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128250": {
+      "content": "<|reserved_special_token_242|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128251": {
+      "content": "<|reserved_special_token_243|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128252": {
+      "content": "<|reserved_special_token_244|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128253": {
+      "content": "<|reserved_special_token_245|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128254": {
+      "content": "<|reserved_special_token_246|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128255": {
+      "content": "<|reserved_special_token_247|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "bos_token": "<|begin_of_text|>",
+  "clean_up_tokenization_spaces": true,
+  "eos_token": "<|eot_id|>",
+  "extra_special_tokens": {},
+  "model_input_names": [
+    "input_ids",
+    "attention_mask"
+  ],
+  "model_max_length": 131072,
+  "pad_token": "<|finetune_right_pad_id|>",
+  "padding_side": "right",
+  "tokenizer_class": "PreTrainedTokenizerFast",
+  "unk_token": null
+}

experiments/de-en-pt-zh/checkpoint-1000/trainer_state.json

@@ -0,0 +1,1434 @@
+{
+  "best_global_step": null,
+  "best_metric": null,
+  "best_model_checkpoint": null,
+  "epoch": 3.1645569620253164,
+  "eval_steps": 500,
+  "global_step": 1000,
+  "is_hyper_param_search": false,
+  "is_local_process_zero": true,
+  "is_world_process_zero": true,
+  "log_history": [
+    {
+      "epoch": 0.015822784810126583,
+      "grad_norm": 6.080485820770264,
+      "learning_rate": 0.0002,
+      "loss": 2.7599,
+      "step": 5
+    },
+    {
+      "epoch": 0.03164556962025317,
+      "grad_norm": 1.6470410823822021,
+      "learning_rate": 0.00019936548223350255,
+      "loss": 0.8662,
+      "step": 10
+    },
+    {
+      "epoch": 0.04746835443037975,
+      "grad_norm": 1.656543254852295,
+      "learning_rate": 0.00019873096446700508,
+      "loss": 0.5796,
+      "step": 15
+    },
+    {
+      "epoch": 0.06329113924050633,
+      "grad_norm": 0.9287125468254089,
+      "learning_rate": 0.00019809644670050762,
+      "loss": 0.5521,
+      "step": 20
+    },
+    {
+      "epoch": 0.07911392405063292,
+      "grad_norm": 0.9500881433486938,
+      "learning_rate": 0.00019746192893401016,
+      "loss": 0.4879,
+      "step": 25
+    },
+    {
+      "epoch": 0.0949367088607595,
+      "grad_norm": 0.6434516310691833,
+      "learning_rate": 0.00019682741116751272,
+      "loss": 0.467,
+      "step": 30
+    },
+    {
+      "epoch": 0.11075949367088607,
+      "grad_norm": 0.8102421760559082,
+      "learning_rate": 0.00019619289340101523,
+      "loss": 0.4872,
+      "step": 35
+    },
+    {
+      "epoch": 0.12658227848101267,
+      "grad_norm": 0.8061529397964478,
+      "learning_rate": 0.00019555837563451777,
+      "loss": 0.4883,
+      "step": 40
+    },
+    {
+      "epoch": 0.14240506329113925,
+      "grad_norm": 0.8552656769752502,
+      "learning_rate": 0.00019492385786802033,
+      "loss": 0.4634,
+      "step": 45
+    },
+    {
+      "epoch": 0.15822784810126583,
+      "grad_norm": 0.6702130436897278,
+      "learning_rate": 0.00019428934010152284,
+      "loss": 0.4606,
+      "step": 50
+    },
+    {
+      "epoch": 0.17405063291139242,
+      "grad_norm": 0.7621438503265381,
+      "learning_rate": 0.00019365482233502538,
+      "loss": 0.5159,
+      "step": 55
+    },
+    {
+      "epoch": 0.189873417721519,
+      "grad_norm": 0.6256672143936157,
+      "learning_rate": 0.00019302030456852794,
+      "loss": 0.4818,
+      "step": 60
+    },
+    {
+      "epoch": 0.20569620253164558,
+      "grad_norm": 0.6232669353485107,
+      "learning_rate": 0.00019238578680203048,
+      "loss": 0.4926,
+      "step": 65
+    },
+    {
+      "epoch": 0.22151898734177214,
+      "grad_norm": 0.6387396454811096,
+      "learning_rate": 0.000191751269035533,
+      "loss": 0.4841,
+      "step": 70
+    },
+    {
+      "epoch": 0.23734177215189872,
+      "grad_norm": 0.6436880230903625,
+      "learning_rate": 0.00019111675126903555,
+      "loss": 0.4871,
+      "step": 75
+    },
+    {
+      "epoch": 0.25316455696202533,
+      "grad_norm": 0.7336615920066833,
+      "learning_rate": 0.0001904822335025381,
+      "loss": 0.4568,
+      "step": 80
+    },
+    {
+      "epoch": 0.2689873417721519,
+      "grad_norm": 0.7388893961906433,
+      "learning_rate": 0.0001898477157360406,
+      "loss": 0.4553,
+      "step": 85
+    },
+    {
+      "epoch": 0.2848101265822785,
+      "grad_norm": 0.6054999828338623,
+      "learning_rate": 0.00018921319796954316,
+      "loss": 0.4405,
+      "step": 90
+    },
+    {
+      "epoch": 0.30063291139240506,
+      "grad_norm": 0.7911633253097534,
+      "learning_rate": 0.0001885786802030457,
+      "loss": 0.5219,
+      "step": 95
+    },
+    {
+      "epoch": 0.31645569620253167,
+      "grad_norm": 0.5389903783798218,
+      "learning_rate": 0.00018794416243654824,
+      "loss": 0.4525,
+      "step": 100
+    },
+    {
+      "epoch": 0.3322784810126582,
+      "grad_norm": 0.7426465749740601,
+      "learning_rate": 0.00018730964467005077,
+      "loss": 0.4856,
+      "step": 105
+    },
+    {
+      "epoch": 0.34810126582278483,
+      "grad_norm": 0.7140562534332275,
+      "learning_rate": 0.0001866751269035533,
+      "loss": 0.4918,
+      "step": 110
+    },
+    {
+      "epoch": 0.3639240506329114,
+      "grad_norm": 0.624547004699707,
+      "learning_rate": 0.00018604060913705585,
+      "loss": 0.4806,
+      "step": 115
+    },
+    {
+      "epoch": 0.379746835443038,
+      "grad_norm": 0.8368127942085266,
+      "learning_rate": 0.00018540609137055838,
+      "loss": 0.4915,
+      "step": 120
+    },
+    {
+      "epoch": 0.39556962025316456,
+      "grad_norm": 0.6069784164428711,
+      "learning_rate": 0.00018477157360406092,
+      "loss": 0.485,
+      "step": 125
+    },
+    {
+      "epoch": 0.41139240506329117,
+      "grad_norm": 0.6394964456558228,
+      "learning_rate": 0.00018413705583756346,
+      "loss": 0.4815,
+      "step": 130
+    },
+    {
+      "epoch": 0.4272151898734177,
+      "grad_norm": 0.6225919127464294,
+      "learning_rate": 0.000183502538071066,
+      "loss": 0.4744,
+      "step": 135
+    },
+    {
+      "epoch": 0.4430379746835443,
+      "grad_norm": 0.7593483328819275,
+      "learning_rate": 0.00018286802030456853,
+      "loss": 0.4505,
+      "step": 140
+    },
+    {
+      "epoch": 0.4588607594936709,
+      "grad_norm": 0.5810648202896118,
+      "learning_rate": 0.00018223350253807107,
+      "loss": 0.435,
+      "step": 145
+    },
+    {
+      "epoch": 0.47468354430379744,
+      "grad_norm": 0.7975546717643738,
+      "learning_rate": 0.00018159898477157363,
+      "loss": 0.4685,
+      "step": 150
+    },
+    {
+      "epoch": 0.49050632911392406,
+      "grad_norm": 0.6529037356376648,
+      "learning_rate": 0.00018096446700507614,
+      "loss": 0.4089,
+      "step": 155
+    },
+    {
+      "epoch": 0.5063291139240507,
+      "grad_norm": 0.7088711857795715,
+      "learning_rate": 0.00018032994923857868,
+      "loss": 0.4388,
+      "step": 160
+    },
+    {
+      "epoch": 0.5221518987341772,
+      "grad_norm": 0.5964807271957397,
+      "learning_rate": 0.00017969543147208124,
+      "loss": 0.4104,
+      "step": 165
+    },
+    {
+      "epoch": 0.5379746835443038,
+      "grad_norm": 0.6564443707466125,
+      "learning_rate": 0.00017906091370558375,
+      "loss": 0.4717,
+      "step": 170
+    },
+    {
+      "epoch": 0.5537974683544303,
+      "grad_norm": 0.6912603974342346,
+      "learning_rate": 0.0001784263959390863,
+      "loss": 0.4528,
+      "step": 175
+    },
+    {
+      "epoch": 0.569620253164557,
+      "grad_norm": 0.6068799495697021,
+      "learning_rate": 0.00017779187817258885,
+      "loss": 0.4714,
+      "step": 180
+    },
+    {
+      "epoch": 0.5854430379746836,
+      "grad_norm": 0.6037777066230774,
+      "learning_rate": 0.0001771573604060914,
+      "loss": 0.5058,
+      "step": 185
+    },
+    {
+      "epoch": 0.6012658227848101,
+      "grad_norm": 0.586581826210022,
+      "learning_rate": 0.0001765228426395939,
+      "loss": 0.439,
+      "step": 190
+    },
+    {
+      "epoch": 0.6170886075949367,
+      "grad_norm": 0.6780263185501099,
+      "learning_rate": 0.00017588832487309646,
+      "loss": 0.4269,
+      "step": 195
+    },
+    {
+      "epoch": 0.6329113924050633,
+      "grad_norm": 0.7488175630569458,
+      "learning_rate": 0.000175253807106599,
+      "loss": 0.4481,
+      "step": 200
+    },
+    {
+      "epoch": 0.6487341772151899,
+      "grad_norm": 0.6138256788253784,
+      "learning_rate": 0.0001746192893401015,
+      "loss": 0.4654,
+      "step": 205
+    },
+    {
+      "epoch": 0.6645569620253164,
+      "grad_norm": 0.7444900274276733,
+      "learning_rate": 0.00017398477157360407,
+      "loss": 0.4165,
+      "step": 210
+    },
+    {
+      "epoch": 0.680379746835443,
+      "grad_norm": 0.6231881380081177,
+      "learning_rate": 0.0001733502538071066,
+      "loss": 0.4469,
+      "step": 215
+    },
+    {
+      "epoch": 0.6962025316455697,
+      "grad_norm": 0.6703267097473145,
+      "learning_rate": 0.00017271573604060915,
+      "loss": 0.4279,
+      "step": 220
+    },
+    {
+      "epoch": 0.7120253164556962,
+      "grad_norm": 0.4783850312232971,
+      "learning_rate": 0.00017208121827411168,
+      "loss": 0.4654,
+      "step": 225
+    },
+    {
+      "epoch": 0.7278481012658228,
+      "grad_norm": 0.7536376118659973,
+      "learning_rate": 0.00017144670050761422,
+      "loss": 0.4537,
+      "step": 230
+    },
+    {
+      "epoch": 0.7436708860759493,
+      "grad_norm": 0.7595057487487793,
+      "learning_rate": 0.00017081218274111676,
+      "loss": 0.486,
+      "step": 235
+    },
+    {
+      "epoch": 0.759493670886076,
+      "grad_norm": 0.8007245659828186,
+      "learning_rate": 0.0001701776649746193,
+      "loss": 0.3919,
+      "step": 240
+    },
+    {
+      "epoch": 0.7753164556962026,
+      "grad_norm": 0.6637867093086243,
+      "learning_rate": 0.00016954314720812183,
+      "loss": 0.418,
+      "step": 245
+    },
+    {
+      "epoch": 0.7911392405063291,
+      "grad_norm": 0.653685986995697,
+      "learning_rate": 0.00016890862944162437,
+      "loss": 0.4326,
+      "step": 250
+    },
+    {
+      "epoch": 0.8069620253164557,
+      "grad_norm": 0.6465121507644653,
+      "learning_rate": 0.00016827411167512693,
+      "loss": 0.445,
+      "step": 255
+    },
+    {
+      "epoch": 0.8227848101265823,
+      "grad_norm": 0.6836300492286682,
+      "learning_rate": 0.00016763959390862944,
+      "loss": 0.4348,
+      "step": 260
+    },
+    {
+      "epoch": 0.8386075949367089,
+      "grad_norm": 0.7092205882072449,
+      "learning_rate": 0.00016700507614213198,
+      "loss": 0.4081,
+      "step": 265
+    },
+    {
+      "epoch": 0.8544303797468354,
+      "grad_norm": 0.7940587997436523,
+      "learning_rate": 0.00016637055837563454,
+      "loss": 0.4316,
+      "step": 270
+    },
+    {
+      "epoch": 0.870253164556962,
+      "grad_norm": 0.6765463352203369,
+      "learning_rate": 0.00016573604060913705,
+      "loss": 0.45,
+      "step": 275
+    },
+    {
+      "epoch": 0.8860759493670886,
+      "grad_norm": 0.8914448022842407,
+      "learning_rate": 0.0001651015228426396,
+      "loss": 0.4202,
+      "step": 280
+    },
+    {
+      "epoch": 0.9018987341772152,
+      "grad_norm": 0.6309791207313538,
+      "learning_rate": 0.00016446700507614215,
+      "loss": 0.4308,
+      "step": 285
+    },
+    {
+      "epoch": 0.9177215189873418,
+      "grad_norm": 0.7343917489051819,
+      "learning_rate": 0.0001638324873096447,
+      "loss": 0.4387,
+      "step": 290
+    },
+    {
+      "epoch": 0.9335443037974683,
+      "grad_norm": 0.5776176452636719,
+      "learning_rate": 0.0001631979695431472,
+      "loss": 0.4239,
+      "step": 295
+    },
+    {
+      "epoch": 0.9493670886075949,
+      "grad_norm": 0.6677407026290894,
+      "learning_rate": 0.00016256345177664976,
+      "loss": 0.4481,
+      "step": 300
+    },
+    {
+      "epoch": 0.9651898734177216,
+      "grad_norm": 0.6818298101425171,
+      "learning_rate": 0.0001619289340101523,
+      "loss": 0.4089,
+      "step": 305
+    },
+    {
+      "epoch": 0.9810126582278481,
+      "grad_norm": 0.9606236219406128,
+      "learning_rate": 0.0001612944162436548,
+      "loss": 0.4491,
+      "step": 310
+    },
+    {
+      "epoch": 0.9968354430379747,
+      "grad_norm": 0.6414772272109985,
+      "learning_rate": 0.00016065989847715737,
+      "loss": 0.4639,
+      "step": 315
+    },
+    {
+      "epoch": 1.0126582278481013,
+      "grad_norm": 0.5724285244941711,
+      "learning_rate": 0.0001600253807106599,
+      "loss": 0.3448,
+      "step": 320
+    },
+    {
+      "epoch": 1.0284810126582278,
+      "grad_norm": 0.8723211884498596,
+      "learning_rate": 0.00015939086294416242,
+      "loss": 0.3483,
+      "step": 325
+    },
+    {
+      "epoch": 1.0443037974683544,
+      "grad_norm": 0.6645333170890808,
+      "learning_rate": 0.00015875634517766498,
+      "loss": 0.3136,
+      "step": 330
+    },
+    {
+      "epoch": 1.0601265822784811,
+      "grad_norm": 0.7657269239425659,
+      "learning_rate": 0.00015812182741116752,
+      "loss": 0.3426,
+      "step": 335
+    },
+    {
+      "epoch": 1.0759493670886076,
+      "grad_norm": 0.6551052927970886,
+      "learning_rate": 0.00015748730964467006,
+      "loss": 0.3482,
+      "step": 340
+    },
+    {
+      "epoch": 1.0917721518987342,
+      "grad_norm": 0.7337777018547058,
+      "learning_rate": 0.0001568527918781726,
+      "loss": 0.3128,
+      "step": 345
+    },
+    {
+      "epoch": 1.1075949367088607,
+      "grad_norm": 0.8943917751312256,
+      "learning_rate": 0.00015621827411167513,
+      "loss": 0.3134,
+      "step": 350
+    },
+    {
+      "epoch": 1.1234177215189873,
+      "grad_norm": 0.71893310546875,
+      "learning_rate": 0.00015558375634517767,
+      "loss": 0.2815,
+      "step": 355
+    },
+    {
+      "epoch": 1.139240506329114,
+      "grad_norm": 0.7433806657791138,
+      "learning_rate": 0.0001549492385786802,
+      "loss": 0.3216,
+      "step": 360
+    },
+    {
+      "epoch": 1.1550632911392404,
+      "grad_norm": 0.7051471471786499,
+      "learning_rate": 0.00015431472081218274,
+      "loss": 0.3719,
+      "step": 365
+    },
+    {
+      "epoch": 1.1708860759493671,
+      "grad_norm": 0.6804415583610535,
+      "learning_rate": 0.00015368020304568528,
+      "loss": 0.3449,
+      "step": 370
+    },
+    {
+      "epoch": 1.1867088607594938,
+      "grad_norm": 0.9129506945610046,
+      "learning_rate": 0.00015304568527918784,
+      "loss": 0.3574,
+      "step": 375
+    },
+    {
+      "epoch": 1.2025316455696202,
+      "grad_norm": 0.8387975692749023,
+      "learning_rate": 0.00015241116751269035,
+      "loss": 0.3321,
+      "step": 380
+    },
+    {
+      "epoch": 1.2183544303797469,
+      "grad_norm": 0.689298152923584,
+      "learning_rate": 0.0001517766497461929,
+      "loss": 0.3156,
+      "step": 385
+    },
+    {
+      "epoch": 1.2341772151898733,
+      "grad_norm": 0.681196928024292,
+      "learning_rate": 0.00015114213197969545,
+      "loss": 0.3371,
+      "step": 390
+    },
+    {
+      "epoch": 1.25,
+      "grad_norm": 0.8151982426643372,
+      "learning_rate": 0.00015050761421319796,
+      "loss": 0.3356,
+      "step": 395
+    },
+    {
+      "epoch": 1.2658227848101267,
+      "grad_norm": 0.789669394493103,
+      "learning_rate": 0.0001498730964467005,
+      "loss": 0.3063,
+      "step": 400
+    },
+    {
+      "epoch": 1.2816455696202531,
+      "grad_norm": 0.7262426614761353,
+      "learning_rate": 0.00014923857868020306,
+      "loss": 0.3179,
+      "step": 405
+    },
+    {
+      "epoch": 1.2974683544303798,
+      "grad_norm": 0.717400074005127,
+      "learning_rate": 0.0001486040609137056,
+      "loss": 0.3266,
+      "step": 410
+    },
+    {
+      "epoch": 1.3132911392405062,
+      "grad_norm": 0.8459286093711853,
+      "learning_rate": 0.0001479695431472081,
+      "loss": 0.3323,
+      "step": 415
+    },
+    {
+      "epoch": 1.3291139240506329,
+      "grad_norm": 0.831169843673706,
+      "learning_rate": 0.00014733502538071067,
+      "loss": 0.3101,
+      "step": 420
+    },
+    {
+      "epoch": 1.3449367088607596,
+      "grad_norm": 0.6811087727546692,
+      "learning_rate": 0.0001467005076142132,
+      "loss": 0.3166,
+      "step": 425
+    },
+    {
+      "epoch": 1.360759493670886,
+      "grad_norm": 0.813766360282898,
+      "learning_rate": 0.00014606598984771572,
+      "loss": 0.3274,
+      "step": 430
+    },
+    {
+      "epoch": 1.3765822784810127,
+      "grad_norm": 0.7633799314498901,
+      "learning_rate": 0.00014543147208121828,
+      "loss": 0.3351,
+      "step": 435
+    },
+    {
+      "epoch": 1.3924050632911391,
+      "grad_norm": 0.7631050944328308,
+      "learning_rate": 0.00014479695431472082,
+      "loss": 0.3045,
+      "step": 440
+    },
+    {
+      "epoch": 1.4082278481012658,
+      "grad_norm": 0.8526734709739685,
+      "learning_rate": 0.00014416243654822336,
+      "loss": 0.3479,
+      "step": 445
+    },
+    {
+      "epoch": 1.4240506329113924,
+      "grad_norm": 0.7672425508499146,
+      "learning_rate": 0.0001435279187817259,
+      "loss": 0.3302,
+      "step": 450
+    },
+    {
+      "epoch": 1.439873417721519,
+      "grad_norm": 0.8053662776947021,
+      "learning_rate": 0.00014289340101522843,
+      "loss": 0.2994,
+      "step": 455
+    },
+    {
+      "epoch": 1.4556962025316456,
+      "grad_norm": 0.9087522625923157,
+      "learning_rate": 0.00014225888324873097,
+      "loss": 0.3233,
+      "step": 460
+    },
+    {
+      "epoch": 1.4715189873417722,
+      "grad_norm": 0.7690944075584412,
+      "learning_rate": 0.0001416243654822335,
+      "loss": 0.2887,
+      "step": 465
+    },
+    {
+      "epoch": 1.4873417721518987,
+      "grad_norm": 0.8058143854141235,
+      "learning_rate": 0.00014098984771573604,
+      "loss": 0.2982,
+      "step": 470
+    },
+    {
+      "epoch": 1.5031645569620253,
+      "grad_norm": 0.7568528056144714,
+      "learning_rate": 0.00014035532994923858,
+      "loss": 0.3103,
+      "step": 475
+    },
+    {
+      "epoch": 1.518987341772152,
+      "grad_norm": 0.755690336227417,
+      "learning_rate": 0.00013972081218274114,
+      "loss": 0.3009,
+      "step": 480
+    },
+    {
+      "epoch": 1.5348101265822784,
+      "grad_norm": 0.8576112985610962,
+      "learning_rate": 0.00013908629441624365,
+      "loss": 0.3317,
+      "step": 485
+    },
+    {
+      "epoch": 1.5506329113924051,
+      "grad_norm": 0.8156176805496216,
+      "learning_rate": 0.0001384517766497462,
+      "loss": 0.3187,
+      "step": 490
+    },
+    {
+      "epoch": 1.5664556962025316,
+      "grad_norm": 0.8467602729797363,
+      "learning_rate": 0.00013781725888324875,
+      "loss": 0.3155,
+      "step": 495
+    },
+    {
+      "epoch": 1.5822784810126582,
+      "grad_norm": 0.8999772071838379,
+      "learning_rate": 0.00013718274111675126,
+      "loss": 0.3402,
+      "step": 500
+    },
+    {
+      "epoch": 1.5981012658227849,
+      "grad_norm": 0.808548092842102,
+      "learning_rate": 0.0001365482233502538,
+      "loss": 0.296,
+      "step": 505
+    },
+    {
+      "epoch": 1.6139240506329116,
+      "grad_norm": 0.7033184170722961,
+      "learning_rate": 0.00013591370558375636,
+      "loss": 0.3052,
+      "step": 510
+    },
+    {
+      "epoch": 1.629746835443038,
+      "grad_norm": 0.8576298356056213,
+      "learning_rate": 0.00013527918781725887,
+      "loss": 0.3253,
+      "step": 515
+    },
+    {
+      "epoch": 1.6455696202531644,
+      "grad_norm": 0.6722769141197205,
+      "learning_rate": 0.0001346446700507614,
+      "loss": 0.2941,
+      "step": 520
+    },
+    {
+      "epoch": 1.6613924050632911,
+      "grad_norm": 0.6628623008728027,
+      "learning_rate": 0.00013401015228426397,
+      "loss": 0.3016,
+      "step": 525
+    },
+    {
+      "epoch": 1.6772151898734178,
+      "grad_norm": 0.9596195220947266,
+      "learning_rate": 0.0001333756345177665,
+      "loss": 0.3234,
+      "step": 530
+    },
+    {
+      "epoch": 1.6930379746835444,
+      "grad_norm": 0.8452304005622864,
+      "learning_rate": 0.00013274111675126905,
+      "loss": 0.3458,
+      "step": 535
+    },
+    {
+      "epoch": 1.7088607594936709,
+      "grad_norm": 0.7414538860321045,
+      "learning_rate": 0.00013210659898477158,
+      "loss": 0.3133,
+      "step": 540
+    },
+    {
+      "epoch": 1.7246835443037973,
+      "grad_norm": 0.6833527088165283,
+      "learning_rate": 0.00013147208121827412,
+      "loss": 0.3008,
+      "step": 545
+    },
+    {
+      "epoch": 1.740506329113924,
+      "grad_norm": 0.8556643724441528,
+      "learning_rate": 0.00013083756345177666,
+      "loss": 0.3035,
+      "step": 550
+    },
+    {
+      "epoch": 1.7563291139240507,
+      "grad_norm": 0.9071274399757385,
+      "learning_rate": 0.0001302030456852792,
+      "loss": 0.304,
+      "step": 555
+    },
+    {
+      "epoch": 1.7721518987341773,
+      "grad_norm": 1.0122044086456299,
+      "learning_rate": 0.00012956852791878173,
+      "loss": 0.2956,
+      "step": 560
+    },
+    {
+      "epoch": 1.7879746835443038,
+      "grad_norm": 0.9109578132629395,
+      "learning_rate": 0.00012893401015228427,
+      "loss": 0.3101,
+      "step": 565
+    },
+    {
+      "epoch": 1.8037974683544302,
+      "grad_norm": 0.9648177623748779,
+      "learning_rate": 0.0001282994923857868,
+      "loss": 0.2987,
+      "step": 570
+    },
+    {
+      "epoch": 1.8196202531645569,
+      "grad_norm": 0.8060092926025391,
+      "learning_rate": 0.00012766497461928934,
+      "loss": 0.3011,
+      "step": 575
+    },
+    {
+      "epoch": 1.8354430379746836,
+      "grad_norm": 0.7650238871574402,
+      "learning_rate": 0.00012703045685279188,
+      "loss": 0.3207,
+      "step": 580
+    },
+    {
+      "epoch": 1.8512658227848102,
+      "grad_norm": 1.0391228199005127,
+      "learning_rate": 0.00012639593908629442,
+      "loss": 0.3,
+      "step": 585
+    },
+    {
+      "epoch": 1.8670886075949367,
+      "grad_norm": 0.7758641242980957,
+      "learning_rate": 0.00012576142131979695,
+      "loss": 0.3109,
+      "step": 590
+    },
+    {
+      "epoch": 1.8829113924050633,
+      "grad_norm": 0.819521427154541,
+      "learning_rate": 0.0001251269035532995,
+      "loss": 0.2921,
+      "step": 595
+    },
+    {
+      "epoch": 1.8987341772151898,
+      "grad_norm": 0.7439246773719788,
+      "learning_rate": 0.00012449238578680205,
+      "loss": 0.3091,
+      "step": 600
+    },
+    {
+      "epoch": 1.9145569620253164,
+      "grad_norm": 0.8022212386131287,
+      "learning_rate": 0.00012385786802030456,
+      "loss": 0.3252,
+      "step": 605
+    },
+    {
+      "epoch": 1.9303797468354431,
+      "grad_norm": 1.0216495990753174,
+      "learning_rate": 0.0001232233502538071,
+      "loss": 0.3434,
+      "step": 610
+    },
+    {
+      "epoch": 1.9462025316455698,
+      "grad_norm": 0.821251630783081,
+      "learning_rate": 0.00012258883248730966,
+      "loss": 0.325,
+      "step": 615
+    },
+    {
+      "epoch": 1.9620253164556962,
+      "grad_norm": 0.744149386882782,
+      "learning_rate": 0.00012195431472081217,
+      "loss": 0.3286,
+      "step": 620
+    },
+    {
+      "epoch": 1.9778481012658227,
+      "grad_norm": 0.8488931655883789,
+      "learning_rate": 0.00012131979695431472,
+      "loss": 0.3048,
+      "step": 625
+    },
+    {
+      "epoch": 1.9936708860759493,
+      "grad_norm": 0.8845570087432861,
+      "learning_rate": 0.00012068527918781727,
+      "loss": 0.3027,
+      "step": 630
+    },
+    {
+      "epoch": 2.009493670886076,
+      "grad_norm": 0.6925209164619446,
+      "learning_rate": 0.00012005076142131981,
+      "loss": 0.2111,
+      "step": 635
+    },
+    {
+      "epoch": 2.0253164556962027,
+      "grad_norm": 1.101040244102478,
+      "learning_rate": 0.00011941624365482233,
+      "loss": 0.1921,
+      "step": 640
+    },
+    {
+      "epoch": 2.041139240506329,
+      "grad_norm": 1.030375361442566,
+      "learning_rate": 0.00011878172588832489,
+      "loss": 0.1878,
+      "step": 645
+    },
+    {
+      "epoch": 2.0569620253164556,
+      "grad_norm": 0.8349786996841431,
+      "learning_rate": 0.00011814720812182742,
+      "loss": 0.1839,
+      "step": 650
+    },
+    {
+      "epoch": 2.0727848101265822,
+      "grad_norm": 0.7567915320396423,
+      "learning_rate": 0.00011751269035532995,
+      "loss": 0.1978,
+      "step": 655
+    },
+    {
+      "epoch": 2.088607594936709,
+      "grad_norm": 0.9065817594528198,
+      "learning_rate": 0.0001168781725888325,
+      "loss": 0.1904,
+      "step": 660
+    },
+    {
+      "epoch": 2.1044303797468356,
+      "grad_norm": 0.8686302900314331,
+      "learning_rate": 0.00011624365482233503,
+      "loss": 0.1898,
+      "step": 665
+    },
+    {
+      "epoch": 2.1202531645569622,
+      "grad_norm": 0.81605064868927,
+      "learning_rate": 0.00011560913705583758,
+      "loss": 0.186,
+      "step": 670
+    },
+    {
+      "epoch": 2.1360759493670884,
+      "grad_norm": 0.8041939735412598,
+      "learning_rate": 0.0001149746192893401,
+      "loss": 0.1988,
+      "step": 675
+    },
+    {
+      "epoch": 2.151898734177215,
+      "grad_norm": 0.9789828658103943,
+      "learning_rate": 0.00011434010152284264,
+      "loss": 0.1886,
+      "step": 680
+    },
+    {
+      "epoch": 2.1677215189873418,
+      "grad_norm": 0.822130560874939,
+      "learning_rate": 0.0001137055837563452,
+      "loss": 0.1903,
+      "step": 685
+    },
+    {
+      "epoch": 2.1835443037974684,
+      "grad_norm": 0.8661246299743652,
+      "learning_rate": 0.00011307106598984772,
+      "loss": 0.1911,
+      "step": 690
+    },
+    {
+      "epoch": 2.199367088607595,
+      "grad_norm": 0.814487636089325,
+      "learning_rate": 0.00011243654822335025,
+      "loss": 0.1801,
+      "step": 695
+    },
+    {
+      "epoch": 2.2151898734177213,
+      "grad_norm": 0.9888129830360413,
+      "learning_rate": 0.0001118020304568528,
+      "loss": 0.1859,
+      "step": 700
+    },
+    {
+      "epoch": 2.231012658227848,
+      "grad_norm": 0.790763258934021,
+      "learning_rate": 0.00011116751269035533,
+      "loss": 0.1834,
+      "step": 705
+    },
+    {
+      "epoch": 2.2468354430379747,
+      "grad_norm": 0.7045260667800903,
+      "learning_rate": 0.00011053299492385786,
+      "loss": 0.1769,
+      "step": 710
+    },
+    {
+      "epoch": 2.2626582278481013,
+      "grad_norm": 0.8450356125831604,
+      "learning_rate": 0.00010989847715736041,
+      "loss": 0.1912,
+      "step": 715
+    },
+    {
+      "epoch": 2.278481012658228,
+      "grad_norm": 0.9821575284004211,
+      "learning_rate": 0.00010926395939086295,
+      "loss": 0.194,
+      "step": 720
+    },
+    {
+      "epoch": 2.2943037974683547,
+      "grad_norm": 0.827716588973999,
+      "learning_rate": 0.00010862944162436547,
+      "loss": 0.1952,
+      "step": 725
+    },
+    {
+      "epoch": 2.310126582278481,
+      "grad_norm": 1.0735716819763184,
+      "learning_rate": 0.00010799492385786802,
+      "loss": 0.1962,
+      "step": 730
+    },
+    {
+      "epoch": 2.3259493670886076,
+      "grad_norm": 0.759575605392456,
+      "learning_rate": 0.00010736040609137057,
+      "loss": 0.1899,
+      "step": 735
+    },
+    {
+      "epoch": 2.3417721518987342,
+      "grad_norm": 1.0905171632766724,
+      "learning_rate": 0.00010672588832487308,
+      "loss": 0.1836,
+      "step": 740
+    },
+    {
+      "epoch": 2.357594936708861,
+      "grad_norm": 0.9864721894264221,
+      "learning_rate": 0.00010609137055837564,
+      "loss": 0.1988,
+      "step": 745
+    },
+    {
+      "epoch": 2.3734177215189876,
+      "grad_norm": 0.8583397269248962,
+      "learning_rate": 0.00010545685279187819,
+      "loss": 0.2032,
+      "step": 750
+    },
+    {
+      "epoch": 2.3892405063291138,
+      "grad_norm": 0.8750898241996765,
+      "learning_rate": 0.00010482233502538072,
+      "loss": 0.1913,
+      "step": 755
+    },
+    {
+      "epoch": 2.4050632911392404,
+      "grad_norm": 0.8484179377555847,
+      "learning_rate": 0.00010418781725888325,
+      "loss": 0.1932,
+      "step": 760
+    },
+    {
+      "epoch": 2.420886075949367,
+      "grad_norm": 0.8363800048828125,
+      "learning_rate": 0.0001035532994923858,
+      "loss": 0.1961,
+      "step": 765
+    },
+    {
+      "epoch": 2.4367088607594938,
+      "grad_norm": 1.0165894031524658,
+      "learning_rate": 0.00010291878172588833,
+      "loss": 0.1997,
+      "step": 770
+    },
+    {
+      "epoch": 2.4525316455696204,
+      "grad_norm": 0.9139420986175537,
+      "learning_rate": 0.00010228426395939086,
+      "loss": 0.185,
+      "step": 775
+    },
+    {
+      "epoch": 2.4683544303797467,
+      "grad_norm": 0.8798001408576965,
+      "learning_rate": 0.0001016497461928934,
+      "loss": 0.1842,
+      "step": 780
+    },
+    {
+      "epoch": 2.4841772151898733,
+      "grad_norm": 0.8399237394332886,
+      "learning_rate": 0.00010101522842639594,
+      "loss": 0.1759,
+      "step": 785
+    },
+    {
+      "epoch": 2.5,
+      "grad_norm": 0.883194625377655,
+      "learning_rate": 0.0001003807106598985,
+      "loss": 0.2027,
+      "step": 790
+    },
+    {
+      "epoch": 2.5158227848101267,
+      "grad_norm": 1.1090173721313477,
+      "learning_rate": 9.974619289340103e-05,
+      "loss": 0.192,
+      "step": 795
+    },
+    {
+      "epoch": 2.5316455696202533,
+      "grad_norm": 0.8255505561828613,
+      "learning_rate": 9.911167512690355e-05,
+      "loss": 0.1962,
+      "step": 800
+    },
+    {
+      "epoch": 2.5474683544303796,
+      "grad_norm": 1.0160300731658936,
+      "learning_rate": 9.847715736040609e-05,
+      "loss": 0.1932,
+      "step": 805
+    },
+    {
+      "epoch": 2.5632911392405062,
+      "grad_norm": 0.7054086327552795,
+      "learning_rate": 9.784263959390864e-05,
+      "loss": 0.186,
+      "step": 810
+    },
+    {
+      "epoch": 2.579113924050633,
+      "grad_norm": 0.7671273946762085,
+      "learning_rate": 9.720812182741116e-05,
+      "loss": 0.1851,
+      "step": 815
+    },
+    {
+      "epoch": 2.5949367088607596,
+      "grad_norm": 0.9428186416625977,
+      "learning_rate": 9.657360406091371e-05,
+      "loss": 0.1903,
+      "step": 820
+    },
+    {
+      "epoch": 2.6107594936708862,
+      "grad_norm": 0.8303772807121277,
+      "learning_rate": 9.593908629441625e-05,
+      "loss": 0.1877,
+      "step": 825
+    },
+    {
+      "epoch": 2.6265822784810124,
+      "grad_norm": 0.9931467175483704,
+      "learning_rate": 9.530456852791879e-05,
+      "loss": 0.2038,
+      "step": 830
+    },
+    {
+      "epoch": 2.642405063291139,
+      "grad_norm": 0.8338640928268433,
+      "learning_rate": 9.467005076142132e-05,
+      "loss": 0.1782,
+      "step": 835
+    },
+    {
+      "epoch": 2.6582278481012658,
+      "grad_norm": 1.0357074737548828,
+      "learning_rate": 9.403553299492386e-05,
+      "loss": 0.204,
+      "step": 840
+    },
+    {
+      "epoch": 2.6740506329113924,
+      "grad_norm": 0.7227370142936707,
+      "learning_rate": 9.34010152284264e-05,
+      "loss": 0.1871,
+      "step": 845
+    },
+    {
+      "epoch": 2.689873417721519,
+      "grad_norm": 0.8483996987342834,
+      "learning_rate": 9.276649746192894e-05,
+      "loss": 0.1884,
+      "step": 850
+    },
+    {
+      "epoch": 2.7056962025316453,
+      "grad_norm": 0.7692596912384033,
+      "learning_rate": 9.213197969543149e-05,
+      "loss": 0.1987,
+      "step": 855
+    },
+    {
+      "epoch": 2.721518987341772,
+      "grad_norm": 0.9548447728157043,
+      "learning_rate": 9.149746192893401e-05,
+      "loss": 0.1836,
+      "step": 860
+    },
+    {
+      "epoch": 2.7373417721518987,
+      "grad_norm": 0.9363961219787598,
+      "learning_rate": 9.086294416243655e-05,
+      "loss": 0.1815,
+      "step": 865
+    },
+    {
+      "epoch": 2.7531645569620253,
+      "grad_norm": 1.0330653190612793,
+      "learning_rate": 9.02284263959391e-05,
+      "loss": 0.1957,
+      "step": 870
+    },
+    {
+      "epoch": 2.768987341772152,
+      "grad_norm": 0.8239451050758362,
+      "learning_rate": 8.959390862944162e-05,
+      "loss": 0.1945,
+      "step": 875
+    },
+    {
+      "epoch": 2.7848101265822782,
+      "grad_norm": 0.7226342558860779,
+      "learning_rate": 8.895939086294417e-05,
+      "loss": 0.1819,
+      "step": 880
+    },
+    {
+      "epoch": 2.8006329113924053,
+      "grad_norm": 1.2467807531356812,
+      "learning_rate": 8.83248730964467e-05,
+      "loss": 0.1723,
+      "step": 885
+    },
+    {
+      "epoch": 2.8164556962025316,
+      "grad_norm": 0.8140600919723511,
+      "learning_rate": 8.769035532994924e-05,
+      "loss": 0.1898,
+      "step": 890
+    },
+    {
+      "epoch": 2.8322784810126582,
+      "grad_norm": 0.8722625970840454,
+      "learning_rate": 8.705583756345178e-05,
+      "loss": 0.1895,
+      "step": 895
+    },
+    {
+      "epoch": 2.848101265822785,
+      "grad_norm": 1.1079012155532837,
+      "learning_rate": 8.642131979695432e-05,
+      "loss": 0.1999,
+      "step": 900
+    },
+    {
+      "epoch": 2.8639240506329116,
+      "grad_norm": 0.8730645775794983,
+      "learning_rate": 8.578680203045685e-05,
+      "loss": 0.1891,
+      "step": 905
+    },
+    {
+      "epoch": 2.879746835443038,
+      "grad_norm": 0.9077411890029907,
+      "learning_rate": 8.515228426395939e-05,
+      "loss": 0.1756,
+      "step": 910
+    },
+    {
+      "epoch": 2.8955696202531644,
+      "grad_norm": 0.9811768531799316,
+      "learning_rate": 8.451776649746194e-05,
+      "loss": 0.1937,
+      "step": 915
+    },
+    {
+      "epoch": 2.911392405063291,
+      "grad_norm": 0.8190383911132812,
+      "learning_rate": 8.388324873096446e-05,
+      "loss": 0.1947,
+      "step": 920
+    },
+    {
+      "epoch": 2.9272151898734178,
+      "grad_norm": 1.0100188255310059,
+      "learning_rate": 8.324873096446701e-05,
+      "loss": 0.2103,
+      "step": 925
+    },
+    {
+      "epoch": 2.9430379746835444,
+      "grad_norm": 0.7750146389007568,
+      "learning_rate": 8.261421319796955e-05,
+      "loss": 0.1828,
+      "step": 930
+    },
+    {
+      "epoch": 2.958860759493671,
+      "grad_norm": 0.7670167684555054,
+      "learning_rate": 8.197969543147207e-05,
+      "loss": 0.1848,
+      "step": 935
+    },
+    {
+      "epoch": 2.9746835443037973,
+      "grad_norm": 0.7348405718803406,
+      "learning_rate": 8.134517766497463e-05,
+      "loss": 0.1852,
+      "step": 940
+    },
+    {
+      "epoch": 2.990506329113924,
+      "grad_norm": 0.8744247555732727,
+      "learning_rate": 8.071065989847716e-05,
+      "loss": 0.1878,
+      "step": 945
+    },
+    {
+      "epoch": 3.0063291139240507,
+      "grad_norm": 0.5043367743492126,
+      "learning_rate": 8.00761421319797e-05,
+      "loss": 0.153,
+      "step": 950
+    },
+    {
+      "epoch": 3.0221518987341773,
+      "grad_norm": 0.5560520887374878,
+      "learning_rate": 7.944162436548224e-05,
+      "loss": 0.1149,
+      "step": 955
+    },
+    {
+      "epoch": 3.037974683544304,
+      "grad_norm": 0.8911687135696411,
+      "learning_rate": 7.880710659898477e-05,
+      "loss": 0.1172,
+      "step": 960
+    },
+    {
+      "epoch": 3.0537974683544302,
+      "grad_norm": 0.7934268116950989,
+      "learning_rate": 7.817258883248731e-05,
+      "loss": 0.1155,
+      "step": 965
+    },
+    {
+      "epoch": 3.069620253164557,
+      "grad_norm": 0.6288787722587585,
+      "learning_rate": 7.753807106598985e-05,
+      "loss": 0.1079,
+      "step": 970
+    },
+    {
+      "epoch": 3.0854430379746836,
+      "grad_norm": 0.6213780641555786,
+      "learning_rate": 7.69035532994924e-05,
+      "loss": 0.1196,
+      "step": 975
+    },
+    {
+      "epoch": 3.1012658227848102,
+      "grad_norm": 0.8186531066894531,
+      "learning_rate": 7.626903553299492e-05,
+      "loss": 0.1215,
+      "step": 980
+    },
+    {
+      "epoch": 3.117088607594937,
+      "grad_norm": 0.6212114691734314,
+      "learning_rate": 7.563451776649747e-05,
+      "loss": 0.1222,
+      "step": 985
+    },
+    {
+      "epoch": 3.132911392405063,
+      "grad_norm": 0.6861391067504883,
+      "learning_rate": 7.500000000000001e-05,
+      "loss": 0.1142,
+      "step": 990
+    },
+    {
+      "epoch": 3.1487341772151898,
+      "grad_norm": 0.5754695534706116,
+      "learning_rate": 7.436548223350253e-05,
+      "loss": 0.1144,
+      "step": 995
+    },
+    {
+      "epoch": 3.1645569620253164,
+      "grad_norm": 0.9572857022285461,
+      "learning_rate": 7.373096446700508e-05,
+      "loss": 0.1157,
+      "step": 1000
+    }
+  ],
+  "logging_steps": 5,
+  "max_steps": 1580,
+  "num_input_tokens_seen": 0,
+  "num_train_epochs": 5,
+  "save_steps": 500,
+  "stateful_callbacks": {
+    "TrainerControl": {
+      "args": {
+        "should_epoch_stop": false,
+        "should_evaluate": false,
+        "should_log": false,
+        "should_save": true,
+        "should_training_stop": false
+      },
+      "attributes": {}
+    }
+  },
+  "total_flos": 8.683279684499866e+16,
+  "train_batch_size": 16,
+  "trial_name": null,
+  "trial_params": null
+}

experiments/de-en-pt-zh/checkpoint-1000/training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:dc1f965478a30d642e1d3280129a9cb365127c4096496511633d6b72e8ccf15e
+size 6289

experiments/de-en-pt-zh/checkpoint-1500/README.md

@@ -0,0 +1,210 @@
+---
+base_model: unsloth/meta-llama-3.1-8b-instruct-bnb-4bit
+library_name: peft
+pipeline_tag: text-generation
+tags:
+- base_model:adapter:unsloth/meta-llama-3.1-8b-instruct-bnb-4bit
+- lora
+- sft
+- transformers
+- trl
+- unsloth
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.18.1

experiments/de-en-pt-zh/checkpoint-1500/adapter_config.json

@@ -0,0 +1,50 @@
+{
+  "alora_invocation_tokens": null,
+  "alpha_pattern": {},
+  "arrow_config": null,
+  "auto_mapping": {
+    "base_model_class": "LlamaForCausalLM",
+    "parent_library": "transformers.models.llama.modeling_llama",
+    "unsloth_fixed": true
+  },
+  "base_model_name_or_path": "unsloth/meta-llama-3.1-8b-instruct-bnb-4bit",
+  "bias": "none",
+  "corda_config": null,
+  "ensure_weight_tying": false,
+  "eva_config": null,
+  "exclude_modules": null,
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 16,
+  "lora_bias": false,
+  "lora_dropout": 0.1,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "peft_version": "0.18.1",
+  "qalora_group_size": 16,
+  "r": 8,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "gate_proj",
+    "v_proj",
+    "o_proj",
+    "k_proj",
+    "up_proj",
+    "down_proj",
+    "q_proj"
+  ],
+  "target_parameters": null,
+  "task_type": "CAUSAL_LM",
+  "trainable_token_indices": null,
+  "use_dora": false,
+  "use_qalora": false,
+  "use_rslora": true
+}

experiments/de-en-pt-zh/checkpoint-1500/adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:50eb536131a48802399250e219fd00b6ea71a00bcf3351d537e6dd4fe59afdbb
+size 83945296

experiments/de-en-pt-zh/checkpoint-1500/chat_template.jinja

@@ -0,0 +1,109 @@
+{{- bos_token }}
+{%- if custom_tools is defined %}
+    {%- set tools = custom_tools %}
+{%- endif %}
+{%- if not tools_in_user_message is defined %}
+    {%- set tools_in_user_message = true %}
+{%- endif %}
+{%- if not date_string is defined %}
+    {%- set date_string = "26 Jul 2024" %}
+{%- endif %}
+{%- if not tools is defined %}
+    {%- set tools = none %}
+{%- endif %}
+
+{#- This block extracts the system message, so we can slot it into the right place. #}
+{%- if messages[0]['role'] == 'system' %}
+    {%- set system_message = messages[0]['content']|trim %}
+    {%- set messages = messages[1:] %}
+{%- else %}
+    {%- set system_message = "" %}
+{%- endif %}
+
+{#- System message + builtin tools #}
+{{- "<|start_header_id|>system<|end_header_id|>\n\n" }}
+{%- if builtin_tools is defined or tools is not none %}
+    {{- "Environment: ipython\n" }}
+{%- endif %}
+{%- if builtin_tools is defined %}
+    {{- "Tools: " + builtin_tools | reject('equalto', 'code_interpreter') | join(", ") + "\n\n"}}
+{%- endif %}
+{{- "Cutting Knowledge Date: December 2023\n" }}
+{{- "Today Date: " + date_string + "\n\n" }}
+{%- if tools is not none and not tools_in_user_message %}
+    {{- "You have access to the following functions. To call a function, please respond with JSON for a function call." }}
+    {{- 'Respond in the format {"name": function name, "parameters": dictionary of argument name and its value}.' }}
+    {{- "Do not use variables.\n\n" }}
+    {%- for t in tools %}
+        {{- t | tojson(indent=4) }}
+        {{- "\n\n" }}
+    {%- endfor %}
+{%- endif %}
+{{- system_message }}
+{{- "<|eot_id|>" }}
+
+{#- Custom tools are passed in a user message with some extra guidance #}
+{%- if tools_in_user_message and not tools is none %}
+    {#- Extract the first user message so we can plug it in here #}
+    {%- if messages | length != 0 %}
+        {%- set first_user_message = messages[0]['content']|trim %}
+        {%- set messages = messages[1:] %}
+    {%- else %}
+        {{- raise_exception("Cannot put tools in the first user message when there's no first user message!") }}
+{%- endif %}
+    {{- '<|start_header_id|>user<|end_header_id|>\n\n' -}}
+    {{- "Given the following functions, please respond with a JSON for a function call " }}
+    {{- "with its proper arguments that best answers the given prompt.\n\n" }}
+    {{- 'Respond in the format {"name": function name, "parameters": dictionary of argument name and its value}.' }}
+    {{- "Do not use variables.\n\n" }}
+    {%- for t in tools %}
+        {{- t | tojson(indent=4) }}
+        {{- "\n\n" }}
+    {%- endfor %}
+    {{- first_user_message + "<|eot_id|>"}}
+{%- endif %}
+
+{%- for message in messages %}
+    {%- if not (message.role == 'ipython' or message.role == 'tool' or 'tool_calls' in message) %}
+        {{- '<|start_header_id|>' + message['role'] + '<|end_header_id|>\n\n'+ message['content'] | trim + '<|eot_id|>' }}
+    {%- elif 'tool_calls' in message %}
+        {%- if not message.tool_calls|length == 1 %}
+            {{- raise_exception("This model only supports single tool-calls at once!") }}
+        {%- endif %}
+        {%- set tool_call = message.tool_calls[0].function %}
+        {%- if builtin_tools is defined and tool_call.name in builtin_tools %}
+            {{- '<|start_header_id|>assistant<|end_header_id|>\n\n' -}}
+            {{- "<|python_tag|>" + tool_call.name + ".call(" }}
+            {%- for arg_name, arg_val in tool_call.arguments | items %}
+                {{- arg_name + '="' + arg_val + '"' }}
+                {%- if not loop.last %}
+                    {{- ", " }}
+                {%- endif %}
+                {%- endfor %}
+            {{- ")" }}
+        {%- else  %}
+            {{- '<|start_header_id|>assistant<|end_header_id|>\n\n' -}}
+            {{- '{"name": "' + tool_call.name + '", ' }}
+            {{- '"parameters": ' }}
+            {{- tool_call.arguments | tojson }}
+            {{- "}" }}
+        {%- endif %}
+        {%- if builtin_tools is defined %}
+            {#- This means we're in ipython mode #}
+            {{- "<|eom_id|>" }}
+        {%- else %}
+            {{- "<|eot_id|>" }}
+        {%- endif %}
+    {%- elif message.role == "tool" or message.role == "ipython" %}
+        {{- "<|start_header_id|>ipython<|end_header_id|>\n\n" }}
+        {%- if message.content is mapping or message.content is iterable %}
+            {{- message.content | tojson }}
+        {%- else %}
+            {{- message.content }}
+        {%- endif %}
+        {{- "<|eot_id|>" }}
+    {%- endif %}
+{%- endfor %}
+{%- if add_generation_prompt %}
+    {{- '<|start_header_id|>assistant<|end_header_id|>\n\n' }}
+{%- endif %}

experiments/de-en-pt-zh/checkpoint-1500/optimizer.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b514a5abaf55dd7ec34b72a3ebd43a6ff564cc076e1e85206f192beb7fc93c5f
+size 43123429

experiments/de-en-pt-zh/checkpoint-1500/rng_state.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c790abd0eaf70f617b2b2cfddcb2e808817e6bea31b87435b9f860fc0067e13a
+size 14645

experiments/de-en-pt-zh/checkpoint-1500/scheduler.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:220f36a0a4023e86d7b17400d7984cec7692207b558d19ef4561ba0236be8bf8
+size 1465

experiments/de-en-pt-zh/checkpoint-1500/special_tokens_map.json

@@ -0,0 +1,23 @@
+{
+  "bos_token": {
+    "content": "<|begin_of_text|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "<|eot_id|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "<|finetune_right_pad_id|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

experiments/de-en-pt-zh/checkpoint-1500/tokenizer.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6b9e4e7fb171f92fd137b777cc2714bf87d11576700a1dcd7a399e7bbe39537b
+size 17209920

experiments/de-en-pt-zh/checkpoint-1500/tokenizer_config.json

@@ -0,0 +1,2066 @@
+{
+  "add_bos_token": true,
+  "added_tokens_decoder": {
+    "128000": {
+      "content": "<|begin_of_text|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128001": {
+      "content": "<|end_of_text|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128002": {
+      "content": "<|reserved_special_token_0|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128003": {
+      "content": "<|reserved_special_token_1|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128004": {
+      "content": "<|finetune_right_pad_id|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128005": {
+      "content": "<|reserved_special_token_2|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128006": {
+      "content": "<|start_header_id|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128007": {
+      "content": "<|end_header_id|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128008": {
+      "content": "<|eom_id|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128009": {
+      "content": "<|eot_id|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128010": {
+      "content": "<|python_tag|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128011": {
+      "content": "<|reserved_special_token_3|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128012": {
+      "content": "<|reserved_special_token_4|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128013": {
+      "content": "<|reserved_special_token_5|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128014": {
+      "content": "<|reserved_special_token_6|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128015": {
+      "content": "<|reserved_special_token_7|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128016": {
+      "content": "<|reserved_special_token_8|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128017": {
+      "content": "<|reserved_special_token_9|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128018": {
+      "content": "<|reserved_special_token_10|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128019": {
+      "content": "<|reserved_special_token_11|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128020": {
+      "content": "<|reserved_special_token_12|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128021": {
+      "content": "<|reserved_special_token_13|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128022": {
+      "content": "<|reserved_special_token_14|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128023": {
+      "content": "<|reserved_special_token_15|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128024": {
+      "content": "<|reserved_special_token_16|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128025": {
+      "content": "<|reserved_special_token_17|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128026": {
+      "content": "<|reserved_special_token_18|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128027": {
+      "content": "<|reserved_special_token_19|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128028": {
+      "content": "<|reserved_special_token_20|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128029": {
+      "content": "<|reserved_special_token_21|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128030": {
+      "content": "<|reserved_special_token_22|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128031": {
+      "content": "<|reserved_special_token_23|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128032": {
+      "content": "<|reserved_special_token_24|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128033": {
+      "content": "<|reserved_special_token_25|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128034": {
+      "content": "<|reserved_special_token_26|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128035": {
+      "content": "<|reserved_special_token_27|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128036": {
+      "content": "<|reserved_special_token_28|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128037": {
+      "content": "<|reserved_special_token_29|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128038": {
+      "content": "<|reserved_special_token_30|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128039": {
+      "content": "<|reserved_special_token_31|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128040": {
+      "content": "<|reserved_special_token_32|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128041": {
+      "content": "<|reserved_special_token_33|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128042": {
+      "content": "<|reserved_special_token_34|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128043": {
+      "content": "<|reserved_special_token_35|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128044": {
+      "content": "<|reserved_special_token_36|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128045": {
+      "content": "<|reserved_special_token_37|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128046": {
+      "content": "<|reserved_special_token_38|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128047": {
+      "content": "<|reserved_special_token_39|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128048": {
+      "content": "<|reserved_special_token_40|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128049": {
+      "content": "<|reserved_special_token_41|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128050": {
+      "content": "<|reserved_special_token_42|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128051": {
+      "content": "<|reserved_special_token_43|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128052": {
+      "content": "<|reserved_special_token_44|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128053": {
+      "content": "<|reserved_special_token_45|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128054": {
+      "content": "<|reserved_special_token_46|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128055": {
+      "content": "<|reserved_special_token_47|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128056": {
+      "content": "<|reserved_special_token_48|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128057": {
+      "content": "<|reserved_special_token_49|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128058": {
+      "content": "<|reserved_special_token_50|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128059": {
+      "content": "<|reserved_special_token_51|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128060": {
+      "content": "<|reserved_special_token_52|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128061": {
+      "content": "<|reserved_special_token_53|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128062": {
+      "content": "<|reserved_special_token_54|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128063": {
+      "content": "<|reserved_special_token_55|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128064": {
+      "content": "<|reserved_special_token_56|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128065": {
+      "content": "<|reserved_special_token_57|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128066": {
+      "content": "<|reserved_special_token_58|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128067": {
+      "content": "<|reserved_special_token_59|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128068": {
+      "content": "<|reserved_special_token_60|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128069": {
+      "content": "<|reserved_special_token_61|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128070": {
+      "content": "<|reserved_special_token_62|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128071": {
+      "content": "<|reserved_special_token_63|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128072": {
+      "content": "<|reserved_special_token_64|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128073": {
+      "content": "<|reserved_special_token_65|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128074": {
+      "content": "<|reserved_special_token_66|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128075": {
+      "content": "<|reserved_special_token_67|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128076": {
+      "content": "<|reserved_special_token_68|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128077": {
+      "content": "<|reserved_special_token_69|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128078": {
+      "content": "<|reserved_special_token_70|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128079": {
+      "content": "<|reserved_special_token_71|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128080": {
+      "content": "<|reserved_special_token_72|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128081": {
+      "content": "<|reserved_special_token_73|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128082": {
+      "content": "<|reserved_special_token_74|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128083": {
+      "content": "<|reserved_special_token_75|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128084": {
+      "content": "<|reserved_special_token_76|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128085": {
+      "content": "<|reserved_special_token_77|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128086": {
+      "content": "<|reserved_special_token_78|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128087": {
+      "content": "<|reserved_special_token_79|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128088": {
+      "content": "<|reserved_special_token_80|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128089": {
+      "content": "<|reserved_special_token_81|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128090": {
+      "content": "<|reserved_special_token_82|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128091": {
+      "content": "<|reserved_special_token_83|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128092": {
+      "content": "<|reserved_special_token_84|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128093": {
+      "content": "<|reserved_special_token_85|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128094": {
+      "content": "<|reserved_special_token_86|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128095": {
+      "content": "<|reserved_special_token_87|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128096": {
+      "content": "<|reserved_special_token_88|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128097": {
+      "content": "<|reserved_special_token_89|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128098": {
+      "content": "<|reserved_special_token_90|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128099": {
+      "content": "<|reserved_special_token_91|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128100": {
+      "content": "<|reserved_special_token_92|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128101": {
+      "content": "<|reserved_special_token_93|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128102": {
+      "content": "<|reserved_special_token_94|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128103": {
+      "content": "<|reserved_special_token_95|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128104": {
+      "content": "<|reserved_special_token_96|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128105": {
+      "content": "<|reserved_special_token_97|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128106": {
+      "content": "<|reserved_special_token_98|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128107": {
+      "content": "<|reserved_special_token_99|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128108": {
+      "content": "<|reserved_special_token_100|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128109": {
+      "content": "<|reserved_special_token_101|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128110": {
+      "content": "<|reserved_special_token_102|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128111": {
+      "content": "<|reserved_special_token_103|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128112": {
+      "content": "<|reserved_special_token_104|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128113": {
+      "content": "<|reserved_special_token_105|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128114": {
+      "content": "<|reserved_special_token_106|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128115": {
+      "content": "<|reserved_special_token_107|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128116": {
+      "content": "<|reserved_special_token_108|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128117": {
+      "content": "<|reserved_special_token_109|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128118": {
+      "content": "<|reserved_special_token_110|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128119": {
+      "content": "<|reserved_special_token_111|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128120": {
+      "content": "<|reserved_special_token_112|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128121": {
+      "content": "<|reserved_special_token_113|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128122": {
+      "content": "<|reserved_special_token_114|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128123": {
+      "content": "<|reserved_special_token_115|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128124": {
+      "content": "<|reserved_special_token_116|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128125": {
+      "content": "<|reserved_special_token_117|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128126": {
+      "content": "<|reserved_special_token_118|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128127": {
+      "content": "<|reserved_special_token_119|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128128": {
+      "content": "<|reserved_special_token_120|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128129": {
+      "content": "<|reserved_special_token_121|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128130": {
+      "content": "<|reserved_special_token_122|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128131": {
+      "content": "<|reserved_special_token_123|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128132": {
+      "content": "<|reserved_special_token_124|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128133": {
+      "content": "<|reserved_special_token_125|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128134": {
+      "content": "<|reserved_special_token_126|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128135": {
+      "content": "<|reserved_special_token_127|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128136": {
+      "content": "<|reserved_special_token_128|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128137": {
+      "content": "<|reserved_special_token_129|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128138": {
+      "content": "<|reserved_special_token_130|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128139": {
+      "content": "<|reserved_special_token_131|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128140": {
+      "content": "<|reserved_special_token_132|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128141": {
+      "content": "<|reserved_special_token_133|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128142": {
+      "content": "<|reserved_special_token_134|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128143": {
+      "content": "<|reserved_special_token_135|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128144": {
+      "content": "<|reserved_special_token_136|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128145": {
+      "content": "<|reserved_special_token_137|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128146": {
+      "content": "<|reserved_special_token_138|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128147": {
+      "content": "<|reserved_special_token_139|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128148": {
+      "content": "<|reserved_special_token_140|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128149": {
+      "content": "<|reserved_special_token_141|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128150": {
+      "content": "<|reserved_special_token_142|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128151": {
+      "content": "<|reserved_special_token_143|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128152": {
+      "content": "<|reserved_special_token_144|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128153": {
+      "content": "<|reserved_special_token_145|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128154": {
+      "content": "<|reserved_special_token_146|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128155": {
+      "content": "<|reserved_special_token_147|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128156": {
+      "content": "<|reserved_special_token_148|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128157": {
+      "content": "<|reserved_special_token_149|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128158": {
+      "content": "<|reserved_special_token_150|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128159": {
+      "content": "<|reserved_special_token_151|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128160": {
+      "content": "<|reserved_special_token_152|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128161": {
+      "content": "<|reserved_special_token_153|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128162": {
+      "content": "<|reserved_special_token_154|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128163": {
+      "content": "<|reserved_special_token_155|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128164": {
+      "content": "<|reserved_special_token_156|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128165": {
+      "content": "<|reserved_special_token_157|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128166": {
+      "content": "<|reserved_special_token_158|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128167": {
+      "content": "<|reserved_special_token_159|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128168": {
+      "content": "<|reserved_special_token_160|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128169": {
+      "content": "<|reserved_special_token_161|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128170": {
+      "content": "<|reserved_special_token_162|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128171": {
+      "content": "<|reserved_special_token_163|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128172": {
+      "content": "<|reserved_special_token_164|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128173": {
+      "content": "<|reserved_special_token_165|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128174": {
+      "content": "<|reserved_special_token_166|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128175": {
+      "content": "<|reserved_special_token_167|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128176": {
+      "content": "<|reserved_special_token_168|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128177": {
+      "content": "<|reserved_special_token_169|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128178": {
+      "content": "<|reserved_special_token_170|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128179": {
+      "content": "<|reserved_special_token_171|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128180": {
+      "content": "<|reserved_special_token_172|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128181": {
+      "content": "<|reserved_special_token_173|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128182": {
+      "content": "<|reserved_special_token_174|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128183": {
+      "content": "<|reserved_special_token_175|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128184": {
+      "content": "<|reserved_special_token_176|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128185": {
+      "content": "<|reserved_special_token_177|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128186": {
+      "content": "<|reserved_special_token_178|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128187": {
+      "content": "<|reserved_special_token_179|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128188": {
+      "content": "<|reserved_special_token_180|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128189": {
+      "content": "<|reserved_special_token_181|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128190": {
+      "content": "<|reserved_special_token_182|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128191": {
+      "content": "<|reserved_special_token_183|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128192": {
+      "content": "<|reserved_special_token_184|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128193": {
+      "content": "<|reserved_special_token_185|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128194": {
+      "content": "<|reserved_special_token_186|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128195": {
+      "content": "<|reserved_special_token_187|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128196": {
+      "content": "<|reserved_special_token_188|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128197": {
+      "content": "<|reserved_special_token_189|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128198": {
+      "content": "<|reserved_special_token_190|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128199": {
+      "content": "<|reserved_special_token_191|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128200": {
+      "content": "<|reserved_special_token_192|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128201": {
+      "content": "<|reserved_special_token_193|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128202": {
+      "content": "<|reserved_special_token_194|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128203": {
+      "content": "<|reserved_special_token_195|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128204": {
+      "content": "<|reserved_special_token_196|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128205": {
+      "content": "<|reserved_special_token_197|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128206": {
+      "content": "<|reserved_special_token_198|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128207": {
+      "content": "<|reserved_special_token_199|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128208": {
+      "content": "<|reserved_special_token_200|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128209": {
+      "content": "<|reserved_special_token_201|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128210": {
+      "content": "<|reserved_special_token_202|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128211": {
+      "content": "<|reserved_special_token_203|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128212": {
+      "content": "<|reserved_special_token_204|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128213": {
+      "content": "<|reserved_special_token_205|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128214": {
+      "content": "<|reserved_special_token_206|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128215": {
+      "content": "<|reserved_special_token_207|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128216": {
+      "content": "<|reserved_special_token_208|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128217": {
+      "content": "<|reserved_special_token_209|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128218": {
+      "content": "<|reserved_special_token_210|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128219": {
+      "content": "<|reserved_special_token_211|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128220": {
+      "content": "<|reserved_special_token_212|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128221": {
+      "content": "<|reserved_special_token_213|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128222": {
+      "content": "<|reserved_special_token_214|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128223": {
+      "content": "<|reserved_special_token_215|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128224": {
+      "content": "<|reserved_special_token_216|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128225": {
+      "content": "<|reserved_special_token_217|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128226": {
+      "content": "<|reserved_special_token_218|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128227": {
+      "content": "<|reserved_special_token_219|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128228": {
+      "content": "<|reserved_special_token_220|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128229": {
+      "content": "<|reserved_special_token_221|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128230": {
+      "content": "<|reserved_special_token_222|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128231": {
+      "content": "<|reserved_special_token_223|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128232": {
+      "content": "<|reserved_special_token_224|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128233": {
+      "content": "<|reserved_special_token_225|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128234": {
+      "content": "<|reserved_special_token_226|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128235": {
+      "content": "<|reserved_special_token_227|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128236": {
+      "content": "<|reserved_special_token_228|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128237": {
+      "content": "<|reserved_special_token_229|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128238": {
+      "content": "<|reserved_special_token_230|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128239": {
+      "content": "<|reserved_special_token_231|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128240": {
+      "content": "<|reserved_special_token_232|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128241": {
+      "content": "<|reserved_special_token_233|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128242": {
+      "content": "<|reserved_special_token_234|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128243": {
+      "content": "<|reserved_special_token_235|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128244": {
+      "content": "<|reserved_special_token_236|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128245": {
+      "content": "<|reserved_special_token_237|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128246": {
+      "content": "<|reserved_special_token_238|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128247": {
+      "content": "<|reserved_special_token_239|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128248": {
+      "content": "<|reserved_special_token_240|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128249": {
+      "content": "<|reserved_special_token_241|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128250": {
+      "content": "<|reserved_special_token_242|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128251": {
+      "content": "<|reserved_special_token_243|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128252": {
+      "content": "<|reserved_special_token_244|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128253": {
+      "content": "<|reserved_special_token_245|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128254": {
+      "content": "<|reserved_special_token_246|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128255": {
+      "content": "<|reserved_special_token_247|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "bos_token": "<|begin_of_text|>",
+  "clean_up_tokenization_spaces": true,
+  "eos_token": "<|eot_id|>",
+  "extra_special_tokens": {},
+  "model_input_names": [
+    "input_ids",
+    "attention_mask"
+  ],
+  "model_max_length": 131072,
+  "pad_token": "<|finetune_right_pad_id|>",
+  "padding_side": "right",
+  "tokenizer_class": "PreTrainedTokenizerFast",
+  "unk_token": null
+}

experiments/de-en-pt-zh/checkpoint-1500/trainer_state.json

@@ -0,0 +1,2134 @@
+{
+  "best_global_step": null,
+  "best_metric": null,
+  "best_model_checkpoint": null,
+  "epoch": 4.746835443037975,
+  "eval_steps": 500,
+  "global_step": 1500,
+  "is_hyper_param_search": false,
+  "is_local_process_zero": true,
+  "is_world_process_zero": true,
+  "log_history": [
+    {
+      "epoch": 0.015822784810126583,
+      "grad_norm": 6.080485820770264,
+      "learning_rate": 0.0002,
+      "loss": 2.7599,
+      "step": 5
+    },
+    {
+      "epoch": 0.03164556962025317,
+      "grad_norm": 1.6470410823822021,
+      "learning_rate": 0.00019936548223350255,
+      "loss": 0.8662,
+      "step": 10
+    },
+    {
+      "epoch": 0.04746835443037975,
+      "grad_norm": 1.656543254852295,
+      "learning_rate": 0.00019873096446700508,
+      "loss": 0.5796,
+      "step": 15
+    },
+    {
+      "epoch": 0.06329113924050633,
+      "grad_norm": 0.9287125468254089,
+      "learning_rate": 0.00019809644670050762,
+      "loss": 0.5521,
+      "step": 20
+    },
+    {
+      "epoch": 0.07911392405063292,
+      "grad_norm": 0.9500881433486938,
+      "learning_rate": 0.00019746192893401016,
+      "loss": 0.4879,
+      "step": 25
+    },
+    {
+      "epoch": 0.0949367088607595,
+      "grad_norm": 0.6434516310691833,
+      "learning_rate": 0.00019682741116751272,
+      "loss": 0.467,
+      "step": 30
+    },
+    {
+      "epoch": 0.11075949367088607,
+      "grad_norm": 0.8102421760559082,
+      "learning_rate": 0.00019619289340101523,
+      "loss": 0.4872,
+      "step": 35
+    },
+    {
+      "epoch": 0.12658227848101267,
+      "grad_norm": 0.8061529397964478,
+      "learning_rate": 0.00019555837563451777,
+      "loss": 0.4883,
+      "step": 40
+    },
+    {
+      "epoch": 0.14240506329113925,
+      "grad_norm": 0.8552656769752502,
+      "learning_rate": 0.00019492385786802033,
+      "loss": 0.4634,
+      "step": 45
+    },
+    {
+      "epoch": 0.15822784810126583,
+      "grad_norm": 0.6702130436897278,
+      "learning_rate": 0.00019428934010152284,
+      "loss": 0.4606,
+      "step": 50
+    },
+    {
+      "epoch": 0.17405063291139242,
+      "grad_norm": 0.7621438503265381,
+      "learning_rate": 0.00019365482233502538,
+      "loss": 0.5159,
+      "step": 55
+    },
+    {
+      "epoch": 0.189873417721519,
+      "grad_norm": 0.6256672143936157,
+      "learning_rate": 0.00019302030456852794,
+      "loss": 0.4818,
+      "step": 60
+    },
+    {
+      "epoch": 0.20569620253164558,
+      "grad_norm": 0.6232669353485107,
+      "learning_rate": 0.00019238578680203048,
+      "loss": 0.4926,
+      "step": 65
+    },
+    {
+      "epoch": 0.22151898734177214,
+      "grad_norm": 0.6387396454811096,
+      "learning_rate": 0.000191751269035533,
+      "loss": 0.4841,
+      "step": 70
+    },
+    {
+      "epoch": 0.23734177215189872,
+      "grad_norm": 0.6436880230903625,
+      "learning_rate": 0.00019111675126903555,
+      "loss": 0.4871,
+      "step": 75
+    },
+    {
+      "epoch": 0.25316455696202533,
+      "grad_norm": 0.7336615920066833,
+      "learning_rate": 0.0001904822335025381,
+      "loss": 0.4568,
+      "step": 80
+    },
+    {
+      "epoch": 0.2689873417721519,
+      "grad_norm": 0.7388893961906433,
+      "learning_rate": 0.0001898477157360406,
+      "loss": 0.4553,
+      "step": 85
+    },
+    {
+      "epoch": 0.2848101265822785,
+      "grad_norm": 0.6054999828338623,
+      "learning_rate": 0.00018921319796954316,
+      "loss": 0.4405,
+      "step": 90
+    },
+    {
+      "epoch": 0.30063291139240506,
+      "grad_norm": 0.7911633253097534,
+      "learning_rate": 0.0001885786802030457,
+      "loss": 0.5219,
+      "step": 95
+    },
+    {
+      "epoch": 0.31645569620253167,
+      "grad_norm": 0.5389903783798218,
+      "learning_rate": 0.00018794416243654824,
+      "loss": 0.4525,
+      "step": 100
+    },
+    {
+      "epoch": 0.3322784810126582,
+      "grad_norm": 0.7426465749740601,
+      "learning_rate": 0.00018730964467005077,
+      "loss": 0.4856,
+      "step": 105
+    },
+    {
+      "epoch": 0.34810126582278483,
+      "grad_norm": 0.7140562534332275,
+      "learning_rate": 0.0001866751269035533,
+      "loss": 0.4918,
+      "step": 110
+    },
+    {
+      "epoch": 0.3639240506329114,
+      "grad_norm": 0.624547004699707,
+      "learning_rate": 0.00018604060913705585,
+      "loss": 0.4806,
+      "step": 115
+    },
+    {
+      "epoch": 0.379746835443038,
+      "grad_norm": 0.8368127942085266,
+      "learning_rate": 0.00018540609137055838,
+      "loss": 0.4915,
+      "step": 120
+    },
+    {
+      "epoch": 0.39556962025316456,
+      "grad_norm": 0.6069784164428711,
+      "learning_rate": 0.00018477157360406092,
+      "loss": 0.485,
+      "step": 125
+    },
+    {
+      "epoch": 0.41139240506329117,
+      "grad_norm": 0.6394964456558228,
+      "learning_rate": 0.00018413705583756346,
+      "loss": 0.4815,
+      "step": 130
+    },
+    {
+      "epoch": 0.4272151898734177,
+      "grad_norm": 0.6225919127464294,
+      "learning_rate": 0.000183502538071066,
+      "loss": 0.4744,
+      "step": 135
+    },
+    {
+      "epoch": 0.4430379746835443,
+      "grad_norm": 0.7593483328819275,
+      "learning_rate": 0.00018286802030456853,
+      "loss": 0.4505,
+      "step": 140
+    },
+    {
+      "epoch": 0.4588607594936709,
+      "grad_norm": 0.5810648202896118,
+      "learning_rate": 0.00018223350253807107,
+      "loss": 0.435,
+      "step": 145
+    },
+    {
+      "epoch": 0.47468354430379744,
+      "grad_norm": 0.7975546717643738,
+      "learning_rate": 0.00018159898477157363,
+      "loss": 0.4685,
+      "step": 150
+    },
+    {
+      "epoch": 0.49050632911392406,
+      "grad_norm": 0.6529037356376648,
+      "learning_rate": 0.00018096446700507614,
+      "loss": 0.4089,
+      "step": 155
+    },
+    {
+      "epoch": 0.5063291139240507,
+      "grad_norm": 0.7088711857795715,
+      "learning_rate": 0.00018032994923857868,
+      "loss": 0.4388,
+      "step": 160
+    },
+    {
+      "epoch": 0.5221518987341772,
+      "grad_norm": 0.5964807271957397,
+      "learning_rate": 0.00017969543147208124,
+      "loss": 0.4104,
+      "step": 165
+    },
+    {
+      "epoch": 0.5379746835443038,
+      "grad_norm": 0.6564443707466125,
+      "learning_rate": 0.00017906091370558375,
+      "loss": 0.4717,
+      "step": 170
+    },
+    {
+      "epoch": 0.5537974683544303,
+      "grad_norm": 0.6912603974342346,
+      "learning_rate": 0.0001784263959390863,
+      "loss": 0.4528,
+      "step": 175
+    },
+    {
+      "epoch": 0.569620253164557,
+      "grad_norm": 0.6068799495697021,
+      "learning_rate": 0.00017779187817258885,
+      "loss": 0.4714,
+      "step": 180
+    },
+    {
+      "epoch": 0.5854430379746836,
+      "grad_norm": 0.6037777066230774,
+      "learning_rate": 0.0001771573604060914,
+      "loss": 0.5058,
+      "step": 185
+    },
+    {
+      "epoch": 0.6012658227848101,
+      "grad_norm": 0.586581826210022,
+      "learning_rate": 0.0001765228426395939,
+      "loss": 0.439,
+      "step": 190
+    },
+    {
+      "epoch": 0.6170886075949367,
+      "grad_norm": 0.6780263185501099,
+      "learning_rate": 0.00017588832487309646,
+      "loss": 0.4269,
+      "step": 195
+    },
+    {
+      "epoch": 0.6329113924050633,
+      "grad_norm": 0.7488175630569458,
+      "learning_rate": 0.000175253807106599,
+      "loss": 0.4481,
+      "step": 200
+    },
+    {
+      "epoch": 0.6487341772151899,
+      "grad_norm": 0.6138256788253784,
+      "learning_rate": 0.0001746192893401015,
+      "loss": 0.4654,
+      "step": 205
+    },
+    {
+      "epoch": 0.6645569620253164,
+      "grad_norm": 0.7444900274276733,
+      "learning_rate": 0.00017398477157360407,
+      "loss": 0.4165,
+      "step": 210
+    },
+    {
+      "epoch": 0.680379746835443,
+      "grad_norm": 0.6231881380081177,
+      "learning_rate": 0.0001733502538071066,
+      "loss": 0.4469,
+      "step": 215
+    },
+    {
+      "epoch": 0.6962025316455697,
+      "grad_norm": 0.6703267097473145,
+      "learning_rate": 0.00017271573604060915,
+      "loss": 0.4279,
+      "step": 220
+    },
+    {
+      "epoch": 0.7120253164556962,
+      "grad_norm": 0.4783850312232971,
+      "learning_rate": 0.00017208121827411168,
+      "loss": 0.4654,
+      "step": 225
+    },
+    {
+      "epoch": 0.7278481012658228,
+      "grad_norm": 0.7536376118659973,
+      "learning_rate": 0.00017144670050761422,
+      "loss": 0.4537,
+      "step": 230
+    },
+    {
+      "epoch": 0.7436708860759493,
+      "grad_norm": 0.7595057487487793,
+      "learning_rate": 0.00017081218274111676,
+      "loss": 0.486,
+      "step": 235
+    },
+    {
+      "epoch": 0.759493670886076,
+      "grad_norm": 0.8007245659828186,
+      "learning_rate": 0.0001701776649746193,
+      "loss": 0.3919,
+      "step": 240
+    },
+    {
+      "epoch": 0.7753164556962026,
+      "grad_norm": 0.6637867093086243,
+      "learning_rate": 0.00016954314720812183,
+      "loss": 0.418,
+      "step": 245
+    },
+    {
+      "epoch": 0.7911392405063291,
+      "grad_norm": 0.653685986995697,
+      "learning_rate": 0.00016890862944162437,
+      "loss": 0.4326,
+      "step": 250
+    },
+    {
+      "epoch": 0.8069620253164557,
+      "grad_norm": 0.6465121507644653,
+      "learning_rate": 0.00016827411167512693,
+      "loss": 0.445,
+      "step": 255
+    },
+    {
+      "epoch": 0.8227848101265823,
+      "grad_norm": 0.6836300492286682,
+      "learning_rate": 0.00016763959390862944,
+      "loss": 0.4348,
+      "step": 260
+    },
+    {
+      "epoch": 0.8386075949367089,
+      "grad_norm": 0.7092205882072449,
+      "learning_rate": 0.00016700507614213198,
+      "loss": 0.4081,
+      "step": 265
+    },
+    {
+      "epoch": 0.8544303797468354,
+      "grad_norm": 0.7940587997436523,
+      "learning_rate": 0.00016637055837563454,
+      "loss": 0.4316,
+      "step": 270
+    },
+    {
+      "epoch": 0.870253164556962,
+      "grad_norm": 0.6765463352203369,
+      "learning_rate": 0.00016573604060913705,
+      "loss": 0.45,
+      "step": 275
+    },
+    {
+      "epoch": 0.8860759493670886,
+      "grad_norm": 0.8914448022842407,
+      "learning_rate": 0.0001651015228426396,
+      "loss": 0.4202,
+      "step": 280
+    },
+    {
+      "epoch": 0.9018987341772152,
+      "grad_norm": 0.6309791207313538,
+      "learning_rate": 0.00016446700507614215,
+      "loss": 0.4308,
+      "step": 285
+    },
+    {
+      "epoch": 0.9177215189873418,
+      "grad_norm": 0.7343917489051819,
+      "learning_rate": 0.0001638324873096447,
+      "loss": 0.4387,
+      "step": 290
+    },
+    {
+      "epoch": 0.9335443037974683,
+      "grad_norm": 0.5776176452636719,
+      "learning_rate": 0.0001631979695431472,
+      "loss": 0.4239,
+      "step": 295
+    },
+    {
+      "epoch": 0.9493670886075949,
+      "grad_norm": 0.6677407026290894,
+      "learning_rate": 0.00016256345177664976,
+      "loss": 0.4481,
+      "step": 300
+    },
+    {
+      "epoch": 0.9651898734177216,
+      "grad_norm": 0.6818298101425171,
+      "learning_rate": 0.0001619289340101523,
+      "loss": 0.4089,
+      "step": 305
+    },
+    {
+      "epoch": 0.9810126582278481,
+      "grad_norm": 0.9606236219406128,
+      "learning_rate": 0.0001612944162436548,
+      "loss": 0.4491,
+      "step": 310
+    },
+    {
+      "epoch": 0.9968354430379747,
+      "grad_norm": 0.6414772272109985,
+      "learning_rate": 0.00016065989847715737,
+      "loss": 0.4639,
+      "step": 315
+    },
+    {
+      "epoch": 1.0126582278481013,
+      "grad_norm": 0.5724285244941711,
+      "learning_rate": 0.0001600253807106599,
+      "loss": 0.3448,
+      "step": 320
+    },
+    {
+      "epoch": 1.0284810126582278,
+      "grad_norm": 0.8723211884498596,
+      "learning_rate": 0.00015939086294416242,
+      "loss": 0.3483,
+      "step": 325
+    },
+    {
+      "epoch": 1.0443037974683544,
+      "grad_norm": 0.6645333170890808,
+      "learning_rate": 0.00015875634517766498,
+      "loss": 0.3136,
+      "step": 330
+    },
+    {
+      "epoch": 1.0601265822784811,
+      "grad_norm": 0.7657269239425659,
+      "learning_rate": 0.00015812182741116752,
+      "loss": 0.3426,
+      "step": 335
+    },
+    {
+      "epoch": 1.0759493670886076,
+      "grad_norm": 0.6551052927970886,
+      "learning_rate": 0.00015748730964467006,
+      "loss": 0.3482,
+      "step": 340
+    },
+    {
+      "epoch": 1.0917721518987342,
+      "grad_norm": 0.7337777018547058,
+      "learning_rate": 0.0001568527918781726,
+      "loss": 0.3128,
+      "step": 345
+    },
+    {
+      "epoch": 1.1075949367088607,
+      "grad_norm": 0.8943917751312256,
+      "learning_rate": 0.00015621827411167513,
+      "loss": 0.3134,
+      "step": 350
+    },
+    {
+      "epoch": 1.1234177215189873,
+      "grad_norm": 0.71893310546875,
+      "learning_rate": 0.00015558375634517767,
+      "loss": 0.2815,
+      "step": 355
+    },
+    {
+      "epoch": 1.139240506329114,
+      "grad_norm": 0.7433806657791138,
+      "learning_rate": 0.0001549492385786802,
+      "loss": 0.3216,
+      "step": 360
+    },
+    {
+      "epoch": 1.1550632911392404,
+      "grad_norm": 0.7051471471786499,
+      "learning_rate": 0.00015431472081218274,
+      "loss": 0.3719,
+      "step": 365
+    },
+    {
+      "epoch": 1.1708860759493671,
+      "grad_norm": 0.6804415583610535,
+      "learning_rate": 0.00015368020304568528,
+      "loss": 0.3449,
+      "step": 370
+    },
+    {
+      "epoch": 1.1867088607594938,
+      "grad_norm": 0.9129506945610046,
+      "learning_rate": 0.00015304568527918784,
+      "loss": 0.3574,
+      "step": 375
+    },
+    {
+      "epoch": 1.2025316455696202,
+      "grad_norm": 0.8387975692749023,
+      "learning_rate": 0.00015241116751269035,
+      "loss": 0.3321,
+      "step": 380
+    },
+    {
+      "epoch": 1.2183544303797469,
+      "grad_norm": 0.689298152923584,
+      "learning_rate": 0.0001517766497461929,
+      "loss": 0.3156,
+      "step": 385
+    },
+    {
+      "epoch": 1.2341772151898733,
+      "grad_norm": 0.681196928024292,
+      "learning_rate": 0.00015114213197969545,
+      "loss": 0.3371,
+      "step": 390
+    },
+    {
+      "epoch": 1.25,
+      "grad_norm": 0.8151982426643372,
+      "learning_rate": 0.00015050761421319796,
+      "loss": 0.3356,
+      "step": 395
+    },
+    {
+      "epoch": 1.2658227848101267,
+      "grad_norm": 0.789669394493103,
+      "learning_rate": 0.0001498730964467005,
+      "loss": 0.3063,
+      "step": 400
+    },
+    {
+      "epoch": 1.2816455696202531,
+      "grad_norm": 0.7262426614761353,
+      "learning_rate": 0.00014923857868020306,
+      "loss": 0.3179,
+      "step": 405
+    },
+    {
+      "epoch": 1.2974683544303798,
+      "grad_norm": 0.717400074005127,
+      "learning_rate": 0.0001486040609137056,
+      "loss": 0.3266,
+      "step": 410
+    },
+    {
+      "epoch": 1.3132911392405062,
+      "grad_norm": 0.8459286093711853,
+      "learning_rate": 0.0001479695431472081,
+      "loss": 0.3323,
+      "step": 415
+    },
+    {
+      "epoch": 1.3291139240506329,
+      "grad_norm": 0.831169843673706,
+      "learning_rate": 0.00014733502538071067,
+      "loss": 0.3101,
+      "step": 420
+    },
+    {
+      "epoch": 1.3449367088607596,
+      "grad_norm": 0.6811087727546692,
+      "learning_rate": 0.0001467005076142132,
+      "loss": 0.3166,
+      "step": 425
+    },
+    {
+      "epoch": 1.360759493670886,
+      "grad_norm": 0.813766360282898,
+      "learning_rate": 0.00014606598984771572,
+      "loss": 0.3274,
+      "step": 430
+    },
+    {
+      "epoch": 1.3765822784810127,
+      "grad_norm": 0.7633799314498901,
+      "learning_rate": 0.00014543147208121828,
+      "loss": 0.3351,
+      "step": 435
+    },
+    {
+      "epoch": 1.3924050632911391,
+      "grad_norm": 0.7631050944328308,
+      "learning_rate": 0.00014479695431472082,
+      "loss": 0.3045,
+      "step": 440
+    },
+    {
+      "epoch": 1.4082278481012658,
+      "grad_norm": 0.8526734709739685,
+      "learning_rate": 0.00014416243654822336,
+      "loss": 0.3479,
+      "step": 445
+    },
+    {
+      "epoch": 1.4240506329113924,
+      "grad_norm": 0.7672425508499146,
+      "learning_rate": 0.0001435279187817259,
+      "loss": 0.3302,
+      "step": 450
+    },
+    {
+      "epoch": 1.439873417721519,
+      "grad_norm": 0.8053662776947021,
+      "learning_rate": 0.00014289340101522843,
+      "loss": 0.2994,
+      "step": 455
+    },
+    {
+      "epoch": 1.4556962025316456,
+      "grad_norm": 0.9087522625923157,
+      "learning_rate": 0.00014225888324873097,
+      "loss": 0.3233,
+      "step": 460
+    },
+    {
+      "epoch": 1.4715189873417722,
+      "grad_norm": 0.7690944075584412,
+      "learning_rate": 0.0001416243654822335,
+      "loss": 0.2887,
+      "step": 465
+    },
+    {
+      "epoch": 1.4873417721518987,
+      "grad_norm": 0.8058143854141235,
+      "learning_rate": 0.00014098984771573604,
+      "loss": 0.2982,
+      "step": 470
+    },
+    {
+      "epoch": 1.5031645569620253,
+      "grad_norm": 0.7568528056144714,
+      "learning_rate": 0.00014035532994923858,
+      "loss": 0.3103,
+      "step": 475
+    },
+    {
+      "epoch": 1.518987341772152,
+      "grad_norm": 0.755690336227417,
+      "learning_rate": 0.00013972081218274114,
+      "loss": 0.3009,
+      "step": 480
+    },
+    {
+      "epoch": 1.5348101265822784,
+      "grad_norm": 0.8576112985610962,
+      "learning_rate": 0.00013908629441624365,
+      "loss": 0.3317,
+      "step": 485
+    },
+    {
+      "epoch": 1.5506329113924051,
+      "grad_norm": 0.8156176805496216,
+      "learning_rate": 0.0001384517766497462,
+      "loss": 0.3187,
+      "step": 490
+    },
+    {
+      "epoch": 1.5664556962025316,
+      "grad_norm": 0.8467602729797363,
+      "learning_rate": 0.00013781725888324875,
+      "loss": 0.3155,
+      "step": 495
+    },
+    {
+      "epoch": 1.5822784810126582,
+      "grad_norm": 0.8999772071838379,
+      "learning_rate": 0.00013718274111675126,
+      "loss": 0.3402,
+      "step": 500
+    },
+    {
+      "epoch": 1.5981012658227849,
+      "grad_norm": 0.808548092842102,
+      "learning_rate": 0.0001365482233502538,
+      "loss": 0.296,
+      "step": 505
+    },
+    {
+      "epoch": 1.6139240506329116,
+      "grad_norm": 0.7033184170722961,
+      "learning_rate": 0.00013591370558375636,
+      "loss": 0.3052,
+      "step": 510
+    },
+    {
+      "epoch": 1.629746835443038,
+      "grad_norm": 0.8576298356056213,
+      "learning_rate": 0.00013527918781725887,
+      "loss": 0.3253,
+      "step": 515
+    },
+    {
+      "epoch": 1.6455696202531644,
+      "grad_norm": 0.6722769141197205,
+      "learning_rate": 0.0001346446700507614,
+      "loss": 0.2941,
+      "step": 520
+    },
+    {
+      "epoch": 1.6613924050632911,
+      "grad_norm": 0.6628623008728027,
+      "learning_rate": 0.00013401015228426397,
+      "loss": 0.3016,
+      "step": 525
+    },
+    {
+      "epoch": 1.6772151898734178,
+      "grad_norm": 0.9596195220947266,
+      "learning_rate": 0.0001333756345177665,
+      "loss": 0.3234,
+      "step": 530
+    },
+    {
+      "epoch": 1.6930379746835444,
+      "grad_norm": 0.8452304005622864,
+      "learning_rate": 0.00013274111675126905,
+      "loss": 0.3458,
+      "step": 535
+    },
+    {
+      "epoch": 1.7088607594936709,
+      "grad_norm": 0.7414538860321045,
+      "learning_rate": 0.00013210659898477158,
+      "loss": 0.3133,
+      "step": 540
+    },
+    {
+      "epoch": 1.7246835443037973,
+      "grad_norm": 0.6833527088165283,
+      "learning_rate": 0.00013147208121827412,
+      "loss": 0.3008,
+      "step": 545
+    },
+    {
+      "epoch": 1.740506329113924,
+      "grad_norm": 0.8556643724441528,
+      "learning_rate": 0.00013083756345177666,
+      "loss": 0.3035,
+      "step": 550
+    },
+    {
+      "epoch": 1.7563291139240507,
+      "grad_norm": 0.9071274399757385,
+      "learning_rate": 0.0001302030456852792,
+      "loss": 0.304,
+      "step": 555
+    },
+    {
+      "epoch": 1.7721518987341773,
+      "grad_norm": 1.0122044086456299,
+      "learning_rate": 0.00012956852791878173,
+      "loss": 0.2956,
+      "step": 560
+    },
+    {
+      "epoch": 1.7879746835443038,
+      "grad_norm": 0.9109578132629395,
+      "learning_rate": 0.00012893401015228427,
+      "loss": 0.3101,
+      "step": 565
+    },
+    {
+      "epoch": 1.8037974683544302,
+      "grad_norm": 0.9648177623748779,
+      "learning_rate": 0.0001282994923857868,
+      "loss": 0.2987,
+      "step": 570
+    },
+    {
+      "epoch": 1.8196202531645569,
+      "grad_norm": 0.8060092926025391,
+      "learning_rate": 0.00012766497461928934,
+      "loss": 0.3011,
+      "step": 575
+    },
+    {
+      "epoch": 1.8354430379746836,
+      "grad_norm": 0.7650238871574402,
+      "learning_rate": 0.00012703045685279188,
+      "loss": 0.3207,
+      "step": 580
+    },
+    {
+      "epoch": 1.8512658227848102,
+      "grad_norm": 1.0391228199005127,
+      "learning_rate": 0.00012639593908629442,
+      "loss": 0.3,
+      "step": 585
+    },
+    {
+      "epoch": 1.8670886075949367,
+      "grad_norm": 0.7758641242980957,
+      "learning_rate": 0.00012576142131979695,
+      "loss": 0.3109,
+      "step": 590
+    },
+    {
+      "epoch": 1.8829113924050633,
+      "grad_norm": 0.819521427154541,
+      "learning_rate": 0.0001251269035532995,
+      "loss": 0.2921,
+      "step": 595
+    },
+    {
+      "epoch": 1.8987341772151898,
+      "grad_norm": 0.7439246773719788,
+      "learning_rate": 0.00012449238578680205,
+      "loss": 0.3091,
+      "step": 600
+    },
+    {
+      "epoch": 1.9145569620253164,
+      "grad_norm": 0.8022212386131287,
+      "learning_rate": 0.00012385786802030456,
+      "loss": 0.3252,
+      "step": 605
+    },
+    {
+      "epoch": 1.9303797468354431,
+      "grad_norm": 1.0216495990753174,
+      "learning_rate": 0.0001232233502538071,
+      "loss": 0.3434,
+      "step": 610
+    },
+    {
+      "epoch": 1.9462025316455698,
+      "grad_norm": 0.821251630783081,
+      "learning_rate": 0.00012258883248730966,
+      "loss": 0.325,
+      "step": 615
+    },
+    {
+      "epoch": 1.9620253164556962,
+      "grad_norm": 0.744149386882782,
+      "learning_rate": 0.00012195431472081217,
+      "loss": 0.3286,
+      "step": 620
+    },
+    {
+      "epoch": 1.9778481012658227,
+      "grad_norm": 0.8488931655883789,
+      "learning_rate": 0.00012131979695431472,
+      "loss": 0.3048,
+      "step": 625
+    },
+    {
+      "epoch": 1.9936708860759493,
+      "grad_norm": 0.8845570087432861,
+      "learning_rate": 0.00012068527918781727,
+      "loss": 0.3027,
+      "step": 630
+    },
+    {
+      "epoch": 2.009493670886076,
+      "grad_norm": 0.6925209164619446,
+      "learning_rate": 0.00012005076142131981,
+      "loss": 0.2111,
+      "step": 635
+    },
+    {
+      "epoch": 2.0253164556962027,
+      "grad_norm": 1.101040244102478,
+      "learning_rate": 0.00011941624365482233,
+      "loss": 0.1921,
+      "step": 640
+    },
+    {
+      "epoch": 2.041139240506329,
+      "grad_norm": 1.030375361442566,
+      "learning_rate": 0.00011878172588832489,
+      "loss": 0.1878,
+      "step": 645
+    },
+    {
+      "epoch": 2.0569620253164556,
+      "grad_norm": 0.8349786996841431,
+      "learning_rate": 0.00011814720812182742,
+      "loss": 0.1839,
+      "step": 650
+    },
+    {
+      "epoch": 2.0727848101265822,
+      "grad_norm": 0.7567915320396423,
+      "learning_rate": 0.00011751269035532995,
+      "loss": 0.1978,
+      "step": 655
+    },
+    {
+      "epoch": 2.088607594936709,
+      "grad_norm": 0.9065817594528198,
+      "learning_rate": 0.0001168781725888325,
+      "loss": 0.1904,
+      "step": 660
+    },
+    {
+      "epoch": 2.1044303797468356,
+      "grad_norm": 0.8686302900314331,
+      "learning_rate": 0.00011624365482233503,
+      "loss": 0.1898,
+      "step": 665
+    },
+    {
+      "epoch": 2.1202531645569622,
+      "grad_norm": 0.81605064868927,
+      "learning_rate": 0.00011560913705583758,
+      "loss": 0.186,
+      "step": 670
+    },
+    {
+      "epoch": 2.1360759493670884,
+      "grad_norm": 0.8041939735412598,
+      "learning_rate": 0.0001149746192893401,
+      "loss": 0.1988,
+      "step": 675
+    },
+    {
+      "epoch": 2.151898734177215,
+      "grad_norm": 0.9789828658103943,
+      "learning_rate": 0.00011434010152284264,
+      "loss": 0.1886,
+      "step": 680
+    },
+    {
+      "epoch": 2.1677215189873418,
+      "grad_norm": 0.822130560874939,
+      "learning_rate": 0.0001137055837563452,
+      "loss": 0.1903,
+      "step": 685
+    },
+    {
+      "epoch": 2.1835443037974684,
+      "grad_norm": 0.8661246299743652,
+      "learning_rate": 0.00011307106598984772,
+      "loss": 0.1911,
+      "step": 690
+    },
+    {
+      "epoch": 2.199367088607595,
+      "grad_norm": 0.814487636089325,
+      "learning_rate": 0.00011243654822335025,
+      "loss": 0.1801,
+      "step": 695
+    },
+    {
+      "epoch": 2.2151898734177213,
+      "grad_norm": 0.9888129830360413,
+      "learning_rate": 0.0001118020304568528,
+      "loss": 0.1859,
+      "step": 700
+    },
+    {
+      "epoch": 2.231012658227848,
+      "grad_norm": 0.790763258934021,
+      "learning_rate": 0.00011116751269035533,
+      "loss": 0.1834,
+      "step": 705
+    },
+    {
+      "epoch": 2.2468354430379747,
+      "grad_norm": 0.7045260667800903,
+      "learning_rate": 0.00011053299492385786,
+      "loss": 0.1769,
+      "step": 710
+    },
+    {
+      "epoch": 2.2626582278481013,
+      "grad_norm": 0.8450356125831604,
+      "learning_rate": 0.00010989847715736041,
+      "loss": 0.1912,
+      "step": 715
+    },
+    {
+      "epoch": 2.278481012658228,
+      "grad_norm": 0.9821575284004211,
+      "learning_rate": 0.00010926395939086295,
+      "loss": 0.194,
+      "step": 720
+    },
+    {
+      "epoch": 2.2943037974683547,
+      "grad_norm": 0.827716588973999,
+      "learning_rate": 0.00010862944162436547,
+      "loss": 0.1952,
+      "step": 725
+    },
+    {
+      "epoch": 2.310126582278481,
+      "grad_norm": 1.0735716819763184,
+      "learning_rate": 0.00010799492385786802,
+      "loss": 0.1962,
+      "step": 730
+    },
+    {
+      "epoch": 2.3259493670886076,
+      "grad_norm": 0.759575605392456,
+      "learning_rate": 0.00010736040609137057,
+      "loss": 0.1899,
+      "step": 735
+    },
+    {
+      "epoch": 2.3417721518987342,
+      "grad_norm": 1.0905171632766724,
+      "learning_rate": 0.00010672588832487308,
+      "loss": 0.1836,
+      "step": 740
+    },
+    {
+      "epoch": 2.357594936708861,
+      "grad_norm": 0.9864721894264221,
+      "learning_rate": 0.00010609137055837564,
+      "loss": 0.1988,
+      "step": 745
+    },
+    {
+      "epoch": 2.3734177215189876,
+      "grad_norm": 0.8583397269248962,
+      "learning_rate": 0.00010545685279187819,
+      "loss": 0.2032,
+      "step": 750
+    },
+    {
+      "epoch": 2.3892405063291138,
+      "grad_norm": 0.8750898241996765,
+      "learning_rate": 0.00010482233502538072,
+      "loss": 0.1913,
+      "step": 755
+    },
+    {
+      "epoch": 2.4050632911392404,
+      "grad_norm": 0.8484179377555847,
+      "learning_rate": 0.00010418781725888325,
+      "loss": 0.1932,
+      "step": 760
+    },
+    {
+      "epoch": 2.420886075949367,
+      "grad_norm": 0.8363800048828125,
+      "learning_rate": 0.0001035532994923858,
+      "loss": 0.1961,
+      "step": 765
+    },
+    {
+      "epoch": 2.4367088607594938,
+      "grad_norm": 1.0165894031524658,
+      "learning_rate": 0.00010291878172588833,
+      "loss": 0.1997,
+      "step": 770
+    },
+    {
+      "epoch": 2.4525316455696204,
+      "grad_norm": 0.9139420986175537,
+      "learning_rate": 0.00010228426395939086,
+      "loss": 0.185,
+      "step": 775
+    },
+    {
+      "epoch": 2.4683544303797467,
+      "grad_norm": 0.8798001408576965,
+      "learning_rate": 0.0001016497461928934,
+      "loss": 0.1842,
+      "step": 780
+    },
+    {
+      "epoch": 2.4841772151898733,
+      "grad_norm": 0.8399237394332886,
+      "learning_rate": 0.00010101522842639594,
+      "loss": 0.1759,
+      "step": 785
+    },
+    {
+      "epoch": 2.5,
+      "grad_norm": 0.883194625377655,
+      "learning_rate": 0.0001003807106598985,
+      "loss": 0.2027,
+      "step": 790
+    },
+    {
+      "epoch": 2.5158227848101267,
+      "grad_norm": 1.1090173721313477,
+      "learning_rate": 9.974619289340103e-05,
+      "loss": 0.192,
+      "step": 795
+    },
+    {
+      "epoch": 2.5316455696202533,
+      "grad_norm": 0.8255505561828613,
+      "learning_rate": 9.911167512690355e-05,
+      "loss": 0.1962,
+      "step": 800
+    },
+    {
+      "epoch": 2.5474683544303796,
+      "grad_norm": 1.0160300731658936,
+      "learning_rate": 9.847715736040609e-05,
+      "loss": 0.1932,
+      "step": 805
+    },
+    {
+      "epoch": 2.5632911392405062,
+      "grad_norm": 0.7054086327552795,
+      "learning_rate": 9.784263959390864e-05,
+      "loss": 0.186,
+      "step": 810
+    },
+    {
+      "epoch": 2.579113924050633,
+      "grad_norm": 0.7671273946762085,
+      "learning_rate": 9.720812182741116e-05,
+      "loss": 0.1851,
+      "step": 815
+    },
+    {
+      "epoch": 2.5949367088607596,
+      "grad_norm": 0.9428186416625977,
+      "learning_rate": 9.657360406091371e-05,
+      "loss": 0.1903,
+      "step": 820
+    },
+    {
+      "epoch": 2.6107594936708862,
+      "grad_norm": 0.8303772807121277,
+      "learning_rate": 9.593908629441625e-05,
+      "loss": 0.1877,
+      "step": 825
+    },
+    {
+      "epoch": 2.6265822784810124,
+      "grad_norm": 0.9931467175483704,
+      "learning_rate": 9.530456852791879e-05,
+      "loss": 0.2038,
+      "step": 830
+    },
+    {
+      "epoch": 2.642405063291139,
+      "grad_norm": 0.8338640928268433,
+      "learning_rate": 9.467005076142132e-05,
+      "loss": 0.1782,
+      "step": 835
+    },
+    {
+      "epoch": 2.6582278481012658,
+      "grad_norm": 1.0357074737548828,
+      "learning_rate": 9.403553299492386e-05,
+      "loss": 0.204,
+      "step": 840
+    },
+    {
+      "epoch": 2.6740506329113924,
+      "grad_norm": 0.7227370142936707,
+      "learning_rate": 9.34010152284264e-05,
+      "loss": 0.1871,
+      "step": 845
+    },
+    {
+      "epoch": 2.689873417721519,
+      "grad_norm": 0.8483996987342834,
+      "learning_rate": 9.276649746192894e-05,
+      "loss": 0.1884,
+      "step": 850
+    },
+    {
+      "epoch": 2.7056962025316453,
+      "grad_norm": 0.7692596912384033,
+      "learning_rate": 9.213197969543149e-05,
+      "loss": 0.1987,
+      "step": 855
+    },
+    {
+      "epoch": 2.721518987341772,
+      "grad_norm": 0.9548447728157043,
+      "learning_rate": 9.149746192893401e-05,
+      "loss": 0.1836,
+      "step": 860
+    },
+    {
+      "epoch": 2.7373417721518987,
+      "grad_norm": 0.9363961219787598,
+      "learning_rate": 9.086294416243655e-05,
+      "loss": 0.1815,
+      "step": 865
+    },
+    {
+      "epoch": 2.7531645569620253,
+      "grad_norm": 1.0330653190612793,
+      "learning_rate": 9.02284263959391e-05,
+      "loss": 0.1957,
+      "step": 870
+    },
+    {
+      "epoch": 2.768987341772152,
+      "grad_norm": 0.8239451050758362,
+      "learning_rate": 8.959390862944162e-05,
+      "loss": 0.1945,
+      "step": 875
+    },
+    {
+      "epoch": 2.7848101265822782,
+      "grad_norm": 0.7226342558860779,
+      "learning_rate": 8.895939086294417e-05,
+      "loss": 0.1819,
+      "step": 880
+    },
+    {
+      "epoch": 2.8006329113924053,
+      "grad_norm": 1.2467807531356812,
+      "learning_rate": 8.83248730964467e-05,
+      "loss": 0.1723,
+      "step": 885
+    },
+    {
+      "epoch": 2.8164556962025316,
+      "grad_norm": 0.8140600919723511,
+      "learning_rate": 8.769035532994924e-05,
+      "loss": 0.1898,
+      "step": 890
+    },
+    {
+      "epoch": 2.8322784810126582,
+      "grad_norm": 0.8722625970840454,
+      "learning_rate": 8.705583756345178e-05,
+      "loss": 0.1895,
+      "step": 895
+    },
+    {
+      "epoch": 2.848101265822785,
+      "grad_norm": 1.1079012155532837,
+      "learning_rate": 8.642131979695432e-05,
+      "loss": 0.1999,
+      "step": 900
+    },
+    {
+      "epoch": 2.8639240506329116,
+      "grad_norm": 0.8730645775794983,
+      "learning_rate": 8.578680203045685e-05,
+      "loss": 0.1891,
+      "step": 905
+    },
+    {
+      "epoch": 2.879746835443038,
+      "grad_norm": 0.9077411890029907,
+      "learning_rate": 8.515228426395939e-05,
+      "loss": 0.1756,
+      "step": 910
+    },
+    {
+      "epoch": 2.8955696202531644,
+      "grad_norm": 0.9811768531799316,
+      "learning_rate": 8.451776649746194e-05,
+      "loss": 0.1937,
+      "step": 915
+    },
+    {
+      "epoch": 2.911392405063291,
+      "grad_norm": 0.8190383911132812,
+      "learning_rate": 8.388324873096446e-05,
+      "loss": 0.1947,
+      "step": 920
+    },
+    {
+      "epoch": 2.9272151898734178,
+      "grad_norm": 1.0100188255310059,
+      "learning_rate": 8.324873096446701e-05,
+      "loss": 0.2103,
+      "step": 925
+    },
+    {
+      "epoch": 2.9430379746835444,
+      "grad_norm": 0.7750146389007568,
+      "learning_rate": 8.261421319796955e-05,
+      "loss": 0.1828,
+      "step": 930
+    },
+    {
+      "epoch": 2.958860759493671,
+      "grad_norm": 0.7670167684555054,
+      "learning_rate": 8.197969543147207e-05,
+      "loss": 0.1848,
+      "step": 935
+    },
+    {
+      "epoch": 2.9746835443037973,
+      "grad_norm": 0.7348405718803406,
+      "learning_rate": 8.134517766497463e-05,
+      "loss": 0.1852,
+      "step": 940
+    },
+    {
+      "epoch": 2.990506329113924,
+      "grad_norm": 0.8744247555732727,
+      "learning_rate": 8.071065989847716e-05,
+      "loss": 0.1878,
+      "step": 945
+    },
+    {
+      "epoch": 3.0063291139240507,
+      "grad_norm": 0.5043367743492126,
+      "learning_rate": 8.00761421319797e-05,
+      "loss": 0.153,
+      "step": 950
+    },
+    {
+      "epoch": 3.0221518987341773,
+      "grad_norm": 0.5560520887374878,
+      "learning_rate": 7.944162436548224e-05,
+      "loss": 0.1149,
+      "step": 955
+    },
+    {
+      "epoch": 3.037974683544304,
+      "grad_norm": 0.8911687135696411,
+      "learning_rate": 7.880710659898477e-05,
+      "loss": 0.1172,
+      "step": 960
+    },
+    {
+      "epoch": 3.0537974683544302,
+      "grad_norm": 0.7934268116950989,
+      "learning_rate": 7.817258883248731e-05,
+      "loss": 0.1155,
+      "step": 965
+    },
+    {
+      "epoch": 3.069620253164557,
+      "grad_norm": 0.6288787722587585,
+      "learning_rate": 7.753807106598985e-05,
+      "loss": 0.1079,
+      "step": 970
+    },
+    {
+      "epoch": 3.0854430379746836,
+      "grad_norm": 0.6213780641555786,
+      "learning_rate": 7.69035532994924e-05,
+      "loss": 0.1196,
+      "step": 975
+    },
+    {
+      "epoch": 3.1012658227848102,
+      "grad_norm": 0.8186531066894531,
+      "learning_rate": 7.626903553299492e-05,
+      "loss": 0.1215,
+      "step": 980
+    },
+    {
+      "epoch": 3.117088607594937,
+      "grad_norm": 0.6212114691734314,
+      "learning_rate": 7.563451776649747e-05,
+      "loss": 0.1222,
+      "step": 985
+    },
+    {
+      "epoch": 3.132911392405063,
+      "grad_norm": 0.6861391067504883,
+      "learning_rate": 7.500000000000001e-05,
+      "loss": 0.1142,
+      "step": 990
+    },
+    {
+      "epoch": 3.1487341772151898,
+      "grad_norm": 0.5754695534706116,
+      "learning_rate": 7.436548223350253e-05,
+      "loss": 0.1144,
+      "step": 995
+    },
+    {
+      "epoch": 3.1645569620253164,
+      "grad_norm": 0.9572857022285461,
+      "learning_rate": 7.373096446700508e-05,
+      "loss": 0.1157,
+      "step": 1000
+    },
+    {
+      "epoch": 3.180379746835443,
+      "grad_norm": 0.6589420437812805,
+      "learning_rate": 7.309644670050762e-05,
+      "loss": 0.1195,
+      "step": 1005
+    },
+    {
+      "epoch": 3.1962025316455698,
+      "grad_norm": 0.7942759990692139,
+      "learning_rate": 7.246192893401015e-05,
+      "loss": 0.1199,
+      "step": 1010
+    },
+    {
+      "epoch": 3.212025316455696,
+      "grad_norm": 0.7023386359214783,
+      "learning_rate": 7.182741116751269e-05,
+      "loss": 0.1198,
+      "step": 1015
+    },
+    {
+      "epoch": 3.2278481012658227,
+      "grad_norm": 0.5077189207077026,
+      "learning_rate": 7.119289340101524e-05,
+      "loss": 0.1193,
+      "step": 1020
+    },
+    {
+      "epoch": 3.2436708860759493,
+      "grad_norm": 1.0440549850463867,
+      "learning_rate": 7.055837563451776e-05,
+      "loss": 0.1141,
+      "step": 1025
+    },
+    {
+      "epoch": 3.259493670886076,
+      "grad_norm": 0.6728818416595459,
+      "learning_rate": 6.99238578680203e-05,
+      "loss": 0.1184,
+      "step": 1030
+    },
+    {
+      "epoch": 3.2753164556962027,
+      "grad_norm": 0.6550622582435608,
+      "learning_rate": 6.928934010152285e-05,
+      "loss": 0.1207,
+      "step": 1035
+    },
+    {
+      "epoch": 3.291139240506329,
+      "grad_norm": 0.9485641121864319,
+      "learning_rate": 6.865482233502538e-05,
+      "loss": 0.1185,
+      "step": 1040
+    },
+    {
+      "epoch": 3.3069620253164556,
+      "grad_norm": 0.7015439867973328,
+      "learning_rate": 6.802030456852793e-05,
+      "loss": 0.1166,
+      "step": 1045
+    },
+    {
+      "epoch": 3.3227848101265822,
+      "grad_norm": 0.7128148674964905,
+      "learning_rate": 6.738578680203046e-05,
+      "loss": 0.1211,
+      "step": 1050
+    },
+    {
+      "epoch": 3.338607594936709,
+      "grad_norm": 0.8616880178451538,
+      "learning_rate": 6.6751269035533e-05,
+      "loss": 0.1224,
+      "step": 1055
+    },
+    {
+      "epoch": 3.3544303797468356,
+      "grad_norm": 1.673384189605713,
+      "learning_rate": 6.611675126903554e-05,
+      "loss": 0.1125,
+      "step": 1060
+    },
+    {
+      "epoch": 3.370253164556962,
+      "grad_norm": 0.7952456474304199,
+      "learning_rate": 6.548223350253807e-05,
+      "loss": 0.1196,
+      "step": 1065
+    },
+    {
+      "epoch": 3.3860759493670884,
+      "grad_norm": 0.7431464791297913,
+      "learning_rate": 6.484771573604061e-05,
+      "loss": 0.1172,
+      "step": 1070
+    },
+    {
+      "epoch": 3.401898734177215,
+      "grad_norm": 0.9536241292953491,
+      "learning_rate": 6.421319796954315e-05,
+      "loss": 0.1227,
+      "step": 1075
+    },
+    {
+      "epoch": 3.4177215189873418,
+      "grad_norm": 0.6027010679244995,
+      "learning_rate": 6.35786802030457e-05,
+      "loss": 0.1183,
+      "step": 1080
+    },
+    {
+      "epoch": 3.4335443037974684,
+      "grad_norm": 0.6765082478523254,
+      "learning_rate": 6.294416243654822e-05,
+      "loss": 0.1222,
+      "step": 1085
+    },
+    {
+      "epoch": 3.449367088607595,
+      "grad_norm": 0.8334203958511353,
+      "learning_rate": 6.230964467005076e-05,
+      "loss": 0.1188,
+      "step": 1090
+    },
+    {
+      "epoch": 3.4651898734177213,
+      "grad_norm": 0.7078428864479065,
+      "learning_rate": 6.167512690355331e-05,
+      "loss": 0.1237,
+      "step": 1095
+    },
+    {
+      "epoch": 3.481012658227848,
+      "grad_norm": 0.5793551206588745,
+      "learning_rate": 6.104060913705584e-05,
+      "loss": 0.1171,
+      "step": 1100
+    },
+    {
+      "epoch": 3.4968354430379747,
+      "grad_norm": 0.816104531288147,
+      "learning_rate": 6.040609137055838e-05,
+      "loss": 0.1216,
+      "step": 1105
+    },
+    {
+      "epoch": 3.5126582278481013,
+      "grad_norm": 0.5165361166000366,
+      "learning_rate": 5.977157360406092e-05,
+      "loss": 0.123,
+      "step": 1110
+    },
+    {
+      "epoch": 3.528481012658228,
+      "grad_norm": 0.7187574505805969,
+      "learning_rate": 5.913705583756346e-05,
+      "loss": 0.1196,
+      "step": 1115
+    },
+    {
+      "epoch": 3.5443037974683547,
+      "grad_norm": 0.8947055339813232,
+      "learning_rate": 5.850253807106599e-05,
+      "loss": 0.121,
+      "step": 1120
+    },
+    {
+      "epoch": 3.560126582278481,
+      "grad_norm": 0.6461635828018188,
+      "learning_rate": 5.786802030456853e-05,
+      "loss": 0.1171,
+      "step": 1125
+    },
+    {
+      "epoch": 3.5759493670886076,
+      "grad_norm": 0.5806897282600403,
+      "learning_rate": 5.723350253807107e-05,
+      "loss": 0.1095,
+      "step": 1130
+    },
+    {
+      "epoch": 3.5917721518987342,
+      "grad_norm": 0.48626071214675903,
+      "learning_rate": 5.65989847715736e-05,
+      "loss": 0.117,
+      "step": 1135
+    },
+    {
+      "epoch": 3.607594936708861,
+      "grad_norm": 0.7649936079978943,
+      "learning_rate": 5.5964467005076145e-05,
+      "loss": 0.1214,
+      "step": 1140
+    },
+    {
+      "epoch": 3.6234177215189876,
+      "grad_norm": 0.8777173757553101,
+      "learning_rate": 5.532994923857868e-05,
+      "loss": 0.126,
+      "step": 1145
+    },
+    {
+      "epoch": 3.6392405063291138,
+      "grad_norm": 0.7628064155578613,
+      "learning_rate": 5.469543147208121e-05,
+      "loss": 0.1194,
+      "step": 1150
+    },
+    {
+      "epoch": 3.6550632911392404,
+      "grad_norm": 0.6170560717582703,
+      "learning_rate": 5.406091370558376e-05,
+      "loss": 0.119,
+      "step": 1155
+    },
+    {
+      "epoch": 3.670886075949367,
+      "grad_norm": 0.6344670057296753,
+      "learning_rate": 5.342639593908629e-05,
+      "loss": 0.1153,
+      "step": 1160
+    },
+    {
+      "epoch": 3.6867088607594938,
+      "grad_norm": 0.6660630702972412,
+      "learning_rate": 5.2791878172588836e-05,
+      "loss": 0.1224,
+      "step": 1165
+    },
+    {
+      "epoch": 3.7025316455696204,
+      "grad_norm": 0.7865625619888306,
+      "learning_rate": 5.215736040609137e-05,
+      "loss": 0.1257,
+      "step": 1170
+    },
+    {
+      "epoch": 3.7183544303797467,
+      "grad_norm": 0.611803412437439,
+      "learning_rate": 5.152284263959392e-05,
+      "loss": 0.1195,
+      "step": 1175
+    },
+    {
+      "epoch": 3.7341772151898733,
+      "grad_norm": 0.9372175931930542,
+      "learning_rate": 5.088832487309645e-05,
+      "loss": 0.1239,
+      "step": 1180
+    },
+    {
+      "epoch": 3.75,
+      "grad_norm": 0.6816895008087158,
+      "learning_rate": 5.0253807106598984e-05,
+      "loss": 0.1228,
+      "step": 1185
+    },
+    {
+      "epoch": 3.7658227848101267,
+      "grad_norm": 0.5107612013816833,
+      "learning_rate": 4.961928934010153e-05,
+      "loss": 0.1171,
+      "step": 1190
+    },
+    {
+      "epoch": 3.7816455696202533,
+      "grad_norm": 0.5778011679649353,
+      "learning_rate": 4.8984771573604064e-05,
+      "loss": 0.1253,
+      "step": 1195
+    },
+    {
+      "epoch": 3.7974683544303796,
+      "grad_norm": 0.7186105251312256,
+      "learning_rate": 4.83502538071066e-05,
+      "loss": 0.1155,
+      "step": 1200
+    },
+    {
+      "epoch": 3.8132911392405062,
+      "grad_norm": 0.7445695996284485,
+      "learning_rate": 4.771573604060914e-05,
+      "loss": 0.1176,
+      "step": 1205
+    },
+    {
+      "epoch": 3.829113924050633,
+      "grad_norm": 0.593574047088623,
+      "learning_rate": 4.7081218274111674e-05,
+      "loss": 0.1158,
+      "step": 1210
+    },
+    {
+      "epoch": 3.8449367088607596,
+      "grad_norm": 0.5334892272949219,
+      "learning_rate": 4.644670050761422e-05,
+      "loss": 0.1172,
+      "step": 1215
+    },
+    {
+      "epoch": 3.8607594936708862,
+      "grad_norm": 0.5709195137023926,
+      "learning_rate": 4.5812182741116755e-05,
+      "loss": 0.1158,
+      "step": 1220
+    },
+    {
+      "epoch": 3.8765822784810124,
+      "grad_norm": 0.6969111561775208,
+      "learning_rate": 4.517766497461929e-05,
+      "loss": 0.1198,
+      "step": 1225
+    },
+    {
+      "epoch": 3.892405063291139,
+      "grad_norm": 0.5967200398445129,
+      "learning_rate": 4.454314720812183e-05,
+      "loss": 0.115,
+      "step": 1230
+    },
+    {
+      "epoch": 3.9082278481012658,
+      "grad_norm": 0.6755451560020447,
+      "learning_rate": 4.3908629441624365e-05,
+      "loss": 0.1171,
+      "step": 1235
+    },
+    {
+      "epoch": 3.9240506329113924,
+      "grad_norm": 0.7690590620040894,
+      "learning_rate": 4.32741116751269e-05,
+      "loss": 0.117,
+      "step": 1240
+    },
+    {
+      "epoch": 3.939873417721519,
+      "grad_norm": 0.7448693513870239,
+      "learning_rate": 4.2639593908629446e-05,
+      "loss": 0.1183,
+      "step": 1245
+    },
+    {
+      "epoch": 3.9556962025316453,
+      "grad_norm": 0.6257800459861755,
+      "learning_rate": 4.200507614213198e-05,
+      "loss": 0.1122,
+      "step": 1250
+    },
+    {
+      "epoch": 3.971518987341772,
+      "grad_norm": 0.7136058807373047,
+      "learning_rate": 4.137055837563452e-05,
+      "loss": 0.1172,
+      "step": 1255
+    },
+    {
+      "epoch": 3.9873417721518987,
+      "grad_norm": 0.8732256293296814,
+      "learning_rate": 4.073604060913706e-05,
+      "loss": 0.1198,
+      "step": 1260
+    },
+    {
+      "epoch": 4.003164556962025,
+      "grad_norm": 0.3908518850803375,
+      "learning_rate": 4.010152284263959e-05,
+      "loss": 0.1182,
+      "step": 1265
+    },
+    {
+      "epoch": 4.018987341772152,
+      "grad_norm": 0.3654375672340393,
+      "learning_rate": 3.946700507614213e-05,
+      "loss": 0.0943,
+      "step": 1270
+    },
+    {
+      "epoch": 4.034810126582278,
+      "grad_norm": 0.44177666306495667,
+      "learning_rate": 3.8832487309644673e-05,
+      "loss": 0.093,
+      "step": 1275
+    },
+    {
+      "epoch": 4.050632911392405,
+      "grad_norm": 0.548538863658905,
+      "learning_rate": 3.819796954314721e-05,
+      "loss": 0.0905,
+      "step": 1280
+    },
+    {
+      "epoch": 4.0664556962025316,
+      "grad_norm": 0.34858939051628113,
+      "learning_rate": 3.756345177664975e-05,
+      "loss": 0.0882,
+      "step": 1285
+    },
+    {
+      "epoch": 4.082278481012658,
+      "grad_norm": 0.44596415758132935,
+      "learning_rate": 3.692893401015229e-05,
+      "loss": 0.091,
+      "step": 1290
+    },
+    {
+      "epoch": 4.098101265822785,
+      "grad_norm": 0.4271854758262634,
+      "learning_rate": 3.629441624365482e-05,
+      "loss": 0.0907,
+      "step": 1295
+    },
+    {
+      "epoch": 4.113924050632911,
+      "grad_norm": 0.7120389938354492,
+      "learning_rate": 3.565989847715736e-05,
+      "loss": 0.0931,
+      "step": 1300
+    },
+    {
+      "epoch": 4.129746835443038,
+      "grad_norm": 0.44612565636634827,
+      "learning_rate": 3.50253807106599e-05,
+      "loss": 0.0887,
+      "step": 1305
+    },
+    {
+      "epoch": 4.1455696202531644,
+      "grad_norm": 0.34034597873687744,
+      "learning_rate": 3.439086294416244e-05,
+      "loss": 0.0903,
+      "step": 1310
+    },
+    {
+      "epoch": 4.1613924050632916,
+      "grad_norm": 0.4155871570110321,
+      "learning_rate": 3.3756345177664975e-05,
+      "loss": 0.092,
+      "step": 1315
+    },
+    {
+      "epoch": 4.177215189873418,
+      "grad_norm": 0.5939737558364868,
+      "learning_rate": 3.312182741116752e-05,
+      "loss": 0.0928,
+      "step": 1320
+    },
+    {
+      "epoch": 4.193037974683544,
+      "grad_norm": 0.611441969871521,
+      "learning_rate": 3.248730964467005e-05,
+      "loss": 0.0897,
+      "step": 1325
+    },
+    {
+      "epoch": 4.208860759493671,
+      "grad_norm": 0.6120615601539612,
+      "learning_rate": 3.185279187817259e-05,
+      "loss": 0.0936,
+      "step": 1330
+    },
+    {
+      "epoch": 4.224683544303797,
+      "grad_norm": 0.4837358593940735,
+      "learning_rate": 3.121827411167513e-05,
+      "loss": 0.0896,
+      "step": 1335
+    },
+    {
+      "epoch": 4.2405063291139244,
+      "grad_norm": 0.40970176458358765,
+      "learning_rate": 3.0583756345177666e-05,
+      "loss": 0.0928,
+      "step": 1340
+    },
+    {
+      "epoch": 4.256329113924051,
+      "grad_norm": 0.4805232286453247,
+      "learning_rate": 2.9949238578680206e-05,
+      "loss": 0.0903,
+      "step": 1345
+    },
+    {
+      "epoch": 4.272151898734177,
+      "grad_norm": 0.4605880379676819,
+      "learning_rate": 2.9314720812182743e-05,
+      "loss": 0.0903,
+      "step": 1350
+    },
+    {
+      "epoch": 4.287974683544304,
+      "grad_norm": 0.4408807158470154,
+      "learning_rate": 2.8680203045685283e-05,
+      "loss": 0.0916,
+      "step": 1355
+    },
+    {
+      "epoch": 4.30379746835443,
+      "grad_norm": 0.4075123071670532,
+      "learning_rate": 2.8045685279187816e-05,
+      "loss": 0.0878,
+      "step": 1360
+    },
+    {
+      "epoch": 4.319620253164557,
+      "grad_norm": 0.5319828987121582,
+      "learning_rate": 2.7411167512690357e-05,
+      "loss": 0.0907,
+      "step": 1365
+    },
+    {
+      "epoch": 4.3354430379746836,
+      "grad_norm": 0.5517973899841309,
+      "learning_rate": 2.6776649746192893e-05,
+      "loss": 0.0895,
+      "step": 1370
+    },
+    {
+      "epoch": 4.35126582278481,
+      "grad_norm": 0.48083606362342834,
+      "learning_rate": 2.6142131979695434e-05,
+      "loss": 0.0921,
+      "step": 1375
+    },
+    {
+      "epoch": 4.367088607594937,
+      "grad_norm": 0.5320701599121094,
+      "learning_rate": 2.5507614213197974e-05,
+      "loss": 0.0917,
+      "step": 1380
+    },
+    {
+      "epoch": 4.382911392405063,
+      "grad_norm": 0.416904479265213,
+      "learning_rate": 2.4873096446700507e-05,
+      "loss": 0.0928,
+      "step": 1385
+    },
+    {
+      "epoch": 4.39873417721519,
+      "grad_norm": 0.49495264887809753,
+      "learning_rate": 2.4238578680203047e-05,
+      "loss": 0.0911,
+      "step": 1390
+    },
+    {
+      "epoch": 4.4145569620253164,
+      "grad_norm": 0.5663470029830933,
+      "learning_rate": 2.3604060913705588e-05,
+      "loss": 0.095,
+      "step": 1395
+    },
+    {
+      "epoch": 4.430379746835443,
+      "grad_norm": 0.49303102493286133,
+      "learning_rate": 2.296954314720812e-05,
+      "loss": 0.0937,
+      "step": 1400
+    },
+    {
+      "epoch": 4.44620253164557,
+      "grad_norm": 0.3795440196990967,
+      "learning_rate": 2.233502538071066e-05,
+      "loss": 0.0899,
+      "step": 1405
+    },
+    {
+      "epoch": 4.462025316455696,
+      "grad_norm": 0.4684379994869232,
+      "learning_rate": 2.17005076142132e-05,
+      "loss": 0.0924,
+      "step": 1410
+    },
+    {
+      "epoch": 4.477848101265823,
+      "grad_norm": 0.3904794156551361,
+      "learning_rate": 2.1065989847715735e-05,
+      "loss": 0.0922,
+      "step": 1415
+    },
+    {
+      "epoch": 4.493670886075949,
+      "grad_norm": 0.6037257313728333,
+      "learning_rate": 2.0431472081218275e-05,
+      "loss": 0.096,
+      "step": 1420
+    },
+    {
+      "epoch": 4.509493670886076,
+      "grad_norm": 0.30922073125839233,
+      "learning_rate": 1.9796954314720815e-05,
+      "loss": 0.091,
+      "step": 1425
+    },
+    {
+      "epoch": 4.525316455696203,
+      "grad_norm": 0.5561403036117554,
+      "learning_rate": 1.916243654822335e-05,
+      "loss": 0.0905,
+      "step": 1430
+    },
+    {
+      "epoch": 4.541139240506329,
+      "grad_norm": 0.389127641916275,
+      "learning_rate": 1.852791878172589e-05,
+      "loss": 0.0906,
+      "step": 1435
+    },
+    {
+      "epoch": 4.556962025316456,
+      "grad_norm": 0.38574787974357605,
+      "learning_rate": 1.789340101522843e-05,
+      "loss": 0.0891,
+      "step": 1440
+    },
+    {
+      "epoch": 4.572784810126582,
+      "grad_norm": 0.42243725061416626,
+      "learning_rate": 1.7258883248730966e-05,
+      "loss": 0.0961,
+      "step": 1445
+    },
+    {
+      "epoch": 4.588607594936709,
+      "grad_norm": 0.522402286529541,
+      "learning_rate": 1.6624365482233503e-05,
+      "loss": 0.0889,
+      "step": 1450
+    },
+    {
+      "epoch": 4.6044303797468356,
+      "grad_norm": 0.3562035858631134,
+      "learning_rate": 1.5989847715736043e-05,
+      "loss": 0.0879,
+      "step": 1455
+    },
+    {
+      "epoch": 4.620253164556962,
+      "grad_norm": 0.7289389967918396,
+      "learning_rate": 1.535532994923858e-05,
+      "loss": 0.0955,
+      "step": 1460
+    },
+    {
+      "epoch": 4.636075949367089,
+      "grad_norm": 0.4322461783885956,
+      "learning_rate": 1.4720812182741117e-05,
+      "loss": 0.0933,
+      "step": 1465
+    },
+    {
+      "epoch": 4.651898734177215,
+      "grad_norm": 0.5375872850418091,
+      "learning_rate": 1.4086294416243657e-05,
+      "loss": 0.0911,
+      "step": 1470
+    },
+    {
+      "epoch": 4.667721518987342,
+      "grad_norm": 0.4194104075431824,
+      "learning_rate": 1.3451776649746192e-05,
+      "loss": 0.0878,
+      "step": 1475
+    },
+    {
+      "epoch": 4.6835443037974684,
+      "grad_norm": 0.4603971242904663,
+      "learning_rate": 1.281725888324873e-05,
+      "loss": 0.0916,
+      "step": 1480
+    },
+    {
+      "epoch": 4.699367088607595,
+      "grad_norm": 0.34578830003738403,
+      "learning_rate": 1.218274111675127e-05,
+      "loss": 0.0905,
+      "step": 1485
+    },
+    {
+      "epoch": 4.715189873417722,
+      "grad_norm": 0.7133288383483887,
+      "learning_rate": 1.1548223350253808e-05,
+      "loss": 0.0889,
+      "step": 1490
+    },
+    {
+      "epoch": 4.731012658227848,
+      "grad_norm": 0.6632990837097168,
+      "learning_rate": 1.0913705583756344e-05,
+      "loss": 0.0912,
+      "step": 1495
+    },
+    {
+      "epoch": 4.746835443037975,
+      "grad_norm": 0.339555561542511,
+      "learning_rate": 1.0279187817258885e-05,
+      "loss": 0.091,
+      "step": 1500
+    }
+  ],
+  "logging_steps": 5,
+  "max_steps": 1580,
+  "num_input_tokens_seen": 0,
+  "num_train_epochs": 5,
+  "save_steps": 500,
+  "stateful_callbacks": {
+    "TrainerControl": {
+      "args": {
+        "should_epoch_stop": false,
+        "should_evaluate": false,
+        "should_log": false,
+        "should_save": true,
+        "should_training_stop": false
+      },
+      "attributes": {}
+    }
+  },
+  "total_flos": 1.3027597240614912e+17,
+  "train_batch_size": 16,
+  "trial_name": null,
+  "trial_params": null
+}

experiments/de-en-pt-zh/checkpoint-1500/training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:dc1f965478a30d642e1d3280129a9cb365127c4096496511633d6b72e8ccf15e
+size 6289

experiments/de-en-pt-zh/checkpoint-1580/README.md

@@ -0,0 +1,210 @@
+---
+base_model: unsloth/meta-llama-3.1-8b-instruct-bnb-4bit
+library_name: peft
+pipeline_tag: text-generation
+tags:
+- base_model:adapter:unsloth/meta-llama-3.1-8b-instruct-bnb-4bit
+- lora
+- sft
+- transformers
+- trl
+- unsloth
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.18.1

experiments/de-en-pt-zh/checkpoint-1580/adapter_config.json

@@ -0,0 +1,50 @@
+{
+  "alora_invocation_tokens": null,
+  "alpha_pattern": {},
+  "arrow_config": null,
+  "auto_mapping": {
+    "base_model_class": "LlamaForCausalLM",
+    "parent_library": "transformers.models.llama.modeling_llama",
+    "unsloth_fixed": true
+  },
+  "base_model_name_or_path": "unsloth/meta-llama-3.1-8b-instruct-bnb-4bit",
+  "bias": "none",
+  "corda_config": null,
+  "ensure_weight_tying": false,
+  "eva_config": null,
+  "exclude_modules": null,
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 16,
+  "lora_bias": false,
+  "lora_dropout": 0.1,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "peft_version": "0.18.1",
+  "qalora_group_size": 16,
+  "r": 8,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "gate_proj",
+    "v_proj",
+    "o_proj",
+    "k_proj",
+    "up_proj",
+    "down_proj",
+    "q_proj"
+  ],
+  "target_parameters": null,
+  "task_type": "CAUSAL_LM",
+  "trainable_token_indices": null,
+  "use_dora": false,
+  "use_qalora": false,
+  "use_rslora": true
+}

experiments/de-en-pt-zh/checkpoint-1580/adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:051126fb9ec33a14e0ca6905264f0cdf6fa6a95969cce1b5cbcee93a06dd58c9
+size 83945296

experiments/de-en-pt-zh/checkpoint-1580/chat_template.jinja

@@ -0,0 +1,109 @@
+{{- bos_token }}
+{%- if custom_tools is defined %}
+    {%- set tools = custom_tools %}
+{%- endif %}
+{%- if not tools_in_user_message is defined %}
+    {%- set tools_in_user_message = true %}
+{%- endif %}
+{%- if not date_string is defined %}
+    {%- set date_string = "26 Jul 2024" %}
+{%- endif %}
+{%- if not tools is defined %}
+    {%- set tools = none %}
+{%- endif %}
+
+{#- This block extracts the system message, so we can slot it into the right place. #}
+{%- if messages[0]['role'] == 'system' %}
+    {%- set system_message = messages[0]['content']|trim %}
+    {%- set messages = messages[1:] %}
+{%- else %}
+    {%- set system_message = "" %}
+{%- endif %}
+
+{#- System message + builtin tools #}
+{{- "<|start_header_id|>system<|end_header_id|>\n\n" }}
+{%- if builtin_tools is defined or tools is not none %}
+    {{- "Environment: ipython\n" }}
+{%- endif %}
+{%- if builtin_tools is defined %}
+    {{- "Tools: " + builtin_tools | reject('equalto', 'code_interpreter') | join(", ") + "\n\n"}}
+{%- endif %}
+{{- "Cutting Knowledge Date: December 2023\n" }}
+{{- "Today Date: " + date_string + "\n\n" }}
+{%- if tools is not none and not tools_in_user_message %}
+    {{- "You have access to the following functions. To call a function, please respond with JSON for a function call." }}
+    {{- 'Respond in the format {"name": function name, "parameters": dictionary of argument name and its value}.' }}
+    {{- "Do not use variables.\n\n" }}
+    {%- for t in tools %}
+        {{- t | tojson(indent=4) }}
+        {{- "\n\n" }}
+    {%- endfor %}
+{%- endif %}
+{{- system_message }}
+{{- "<|eot_id|>" }}
+
+{#- Custom tools are passed in a user message with some extra guidance #}
+{%- if tools_in_user_message and not tools is none %}
+    {#- Extract the first user message so we can plug it in here #}
+    {%- if messages | length != 0 %}
+        {%- set first_user_message = messages[0]['content']|trim %}
+        {%- set messages = messages[1:] %}
+    {%- else %}
+        {{- raise_exception("Cannot put tools in the first user message when there's no first user message!") }}
+{%- endif %}
+    {{- '<|start_header_id|>user<|end_header_id|>\n\n' -}}
+    {{- "Given the following functions, please respond with a JSON for a function call " }}
+    {{- "with its proper arguments that best answers the given prompt.\n\n" }}
+    {{- 'Respond in the format {"name": function name, "parameters": dictionary of argument name and its value}.' }}
+    {{- "Do not use variables.\n\n" }}
+    {%- for t in tools %}
+        {{- t | tojson(indent=4) }}
+        {{- "\n\n" }}
+    {%- endfor %}
+    {{- first_user_message + "<|eot_id|>"}}
+{%- endif %}
+
+{%- for message in messages %}
+    {%- if not (message.role == 'ipython' or message.role == 'tool' or 'tool_calls' in message) %}
+        {{- '<|start_header_id|>' + message['role'] + '<|end_header_id|>\n\n'+ message['content'] | trim + '<|eot_id|>' }}
+    {%- elif 'tool_calls' in message %}
+        {%- if not message.tool_calls|length == 1 %}
+            {{- raise_exception("This model only supports single tool-calls at once!") }}
+        {%- endif %}
+        {%- set tool_call = message.tool_calls[0].function %}
+        {%- if builtin_tools is defined and tool_call.name in builtin_tools %}
+            {{- '<|start_header_id|>assistant<|end_header_id|>\n\n' -}}
+            {{- "<|python_tag|>" + tool_call.name + ".call(" }}
+            {%- for arg_name, arg_val in tool_call.arguments | items %}
+                {{- arg_name + '="' + arg_val + '"' }}
+                {%- if not loop.last %}
+                    {{- ", " }}
+                {%- endif %}
+                {%- endfor %}
+            {{- ")" }}
+        {%- else  %}
+            {{- '<|start_header_id|>assistant<|end_header_id|>\n\n' -}}
+            {{- '{"name": "' + tool_call.name + '", ' }}
+            {{- '"parameters": ' }}
+            {{- tool_call.arguments | tojson }}
+            {{- "}" }}
+        {%- endif %}
+        {%- if builtin_tools is defined %}
+            {#- This means we're in ipython mode #}
+            {{- "<|eom_id|>" }}
+        {%- else %}
+            {{- "<|eot_id|>" }}
+        {%- endif %}
+    {%- elif message.role == "tool" or message.role == "ipython" %}
+        {{- "<|start_header_id|>ipython<|end_header_id|>\n\n" }}
+        {%- if message.content is mapping or message.content is iterable %}
+            {{- message.content | tojson }}
+        {%- else %}
+            {{- message.content }}
+        {%- endif %}
+        {{- "<|eot_id|>" }}
+    {%- endif %}
+{%- endfor %}
+{%- if add_generation_prompt %}
+    {{- '<|start_header_id|>assistant<|end_header_id|>\n\n' }}
+{%- endif %}

experiments/de-en-pt-zh/checkpoint-1580/optimizer.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:36cedc14f9390b752f44508d1a359a92966b3903d06f27c12b7d672c9e38623a
+size 43123429

experiments/de-en-pt-zh/checkpoint-1580/rng_state.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:da18f8889d1a2c7f22914265e4646c514d34fc958c2d5959ecdbd416ac8c634e
+size 14645

experiments/de-en-pt-zh/checkpoint-1580/scheduler.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0a97f21c0a7d8518fbc52270f43024d3897b647f2dc864b2af25dddfb74924be
+size 1465

experiments/de-en-pt-zh/checkpoint-1580/special_tokens_map.json

@@ -0,0 +1,23 @@
+{
+  "bos_token": {
+    "content": "<|begin_of_text|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "<|eot_id|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "<|finetune_right_pad_id|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

experiments/de-en-pt-zh/checkpoint-1580/tokenizer.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6b9e4e7fb171f92fd137b777cc2714bf87d11576700a1dcd7a399e7bbe39537b
+size 17209920

experiments/de-en-pt-zh/checkpoint-1580/tokenizer_config.json

@@ -0,0 +1,2066 @@
+{
+  "add_bos_token": true,
+  "added_tokens_decoder": {
+    "128000": {
+      "content": "<|begin_of_text|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128001": {
+      "content": "<|end_of_text|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128002": {
+      "content": "<|reserved_special_token_0|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128003": {
+      "content": "<|reserved_special_token_1|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128004": {
+      "content": "<|finetune_right_pad_id|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128005": {
+      "content": "<|reserved_special_token_2|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128006": {
+      "content": "<|start_header_id|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128007": {
+      "content": "<|end_header_id|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128008": {
+      "content": "<|eom_id|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128009": {
+      "content": "<|eot_id|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128010": {
+      "content": "<|python_tag|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128011": {
+      "content": "<|reserved_special_token_3|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128012": {
+      "content": "<|reserved_special_token_4|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128013": {
+      "content": "<|reserved_special_token_5|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128014": {
+      "content": "<|reserved_special_token_6|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128015": {
+      "content": "<|reserved_special_token_7|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128016": {
+      "content": "<|reserved_special_token_8|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128017": {
+      "content": "<|reserved_special_token_9|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128018": {
+      "content": "<|reserved_special_token_10|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128019": {
+      "content": "<|reserved_special_token_11|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128020": {
+      "content": "<|reserved_special_token_12|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128021": {
+      "content": "<|reserved_special_token_13|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128022": {
+      "content": "<|reserved_special_token_14|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128023": {
+      "content": "<|reserved_special_token_15|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128024": {
+      "content": "<|reserved_special_token_16|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128025": {
+      "content": "<|reserved_special_token_17|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128026": {
+      "content": "<|reserved_special_token_18|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128027": {
+      "content": "<|reserved_special_token_19|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128028": {
+      "content": "<|reserved_special_token_20|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128029": {
+      "content": "<|reserved_special_token_21|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128030": {
+      "content": "<|reserved_special_token_22|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128031": {
+      "content": "<|reserved_special_token_23|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128032": {
+      "content": "<|reserved_special_token_24|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128033": {
+      "content": "<|reserved_special_token_25|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128034": {
+      "content": "<|reserved_special_token_26|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128035": {
+      "content": "<|reserved_special_token_27|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128036": {
+      "content": "<|reserved_special_token_28|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128037": {
+      "content": "<|reserved_special_token_29|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128038": {
+      "content": "<|reserved_special_token_30|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128039": {
+      "content": "<|reserved_special_token_31|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128040": {
+      "content": "<|reserved_special_token_32|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128041": {
+      "content": "<|reserved_special_token_33|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128042": {
+      "content": "<|reserved_special_token_34|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128043": {
+      "content": "<|reserved_special_token_35|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128044": {
+      "content": "<|reserved_special_token_36|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128045": {
+      "content": "<|reserved_special_token_37|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128046": {
+      "content": "<|reserved_special_token_38|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128047": {
+      "content": "<|reserved_special_token_39|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128048": {
+      "content": "<|reserved_special_token_40|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128049": {
+      "content": "<|reserved_special_token_41|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128050": {
+      "content": "<|reserved_special_token_42|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128051": {
+      "content": "<|reserved_special_token_43|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128052": {
+      "content": "<|reserved_special_token_44|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128053": {
+      "content": "<|reserved_special_token_45|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128054": {
+      "content": "<|reserved_special_token_46|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128055": {
+      "content": "<|reserved_special_token_47|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128056": {
+      "content": "<|reserved_special_token_48|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128057": {
+      "content": "<|reserved_special_token_49|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128058": {
+      "content": "<|reserved_special_token_50|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128059": {
+      "content": "<|reserved_special_token_51|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128060": {
+      "content": "<|reserved_special_token_52|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128061": {
+      "content": "<|reserved_special_token_53|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128062": {
+      "content": "<|reserved_special_token_54|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128063": {
+      "content": "<|reserved_special_token_55|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128064": {
+      "content": "<|reserved_special_token_56|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128065": {
+      "content": "<|reserved_special_token_57|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128066": {
+      "content": "<|reserved_special_token_58|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128067": {
+      "content": "<|reserved_special_token_59|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128068": {
+      "content": "<|reserved_special_token_60|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128069": {
+      "content": "<|reserved_special_token_61|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128070": {
+      "content": "<|reserved_special_token_62|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128071": {
+      "content": "<|reserved_special_token_63|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128072": {
+      "content": "<|reserved_special_token_64|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128073": {
+      "content": "<|reserved_special_token_65|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128074": {
+      "content": "<|reserved_special_token_66|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128075": {
+      "content": "<|reserved_special_token_67|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128076": {
+      "content": "<|reserved_special_token_68|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128077": {
+      "content": "<|reserved_special_token_69|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128078": {
+      "content": "<|reserved_special_token_70|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128079": {
+      "content": "<|reserved_special_token_71|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128080": {
+      "content": "<|reserved_special_token_72|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128081": {
+      "content": "<|reserved_special_token_73|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128082": {
+      "content": "<|reserved_special_token_74|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128083": {
+      "content": "<|reserved_special_token_75|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128084": {
+      "content": "<|reserved_special_token_76|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128085": {
+      "content": "<|reserved_special_token_77|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128086": {
+      "content": "<|reserved_special_token_78|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128087": {
+      "content": "<|reserved_special_token_79|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128088": {
+      "content": "<|reserved_special_token_80|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128089": {
+      "content": "<|reserved_special_token_81|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128090": {
+      "content": "<|reserved_special_token_82|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128091": {
+      "content": "<|reserved_special_token_83|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128092": {
+      "content": "<|reserved_special_token_84|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128093": {
+      "content": "<|reserved_special_token_85|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128094": {
+      "content": "<|reserved_special_token_86|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128095": {
+      "content": "<|reserved_special_token_87|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128096": {
+      "content": "<|reserved_special_token_88|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128097": {
+      "content": "<|reserved_special_token_89|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128098": {
+      "content": "<|reserved_special_token_90|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128099": {
+      "content": "<|reserved_special_token_91|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128100": {
+      "content": "<|reserved_special_token_92|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128101": {
+      "content": "<|reserved_special_token_93|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128102": {
+      "content": "<|reserved_special_token_94|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128103": {
+      "content": "<|reserved_special_token_95|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128104": {
+      "content": "<|reserved_special_token_96|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128105": {
+      "content": "<|reserved_special_token_97|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128106": {
+      "content": "<|reserved_special_token_98|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128107": {
+      "content": "<|reserved_special_token_99|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128108": {
+      "content": "<|reserved_special_token_100|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128109": {
+      "content": "<|reserved_special_token_101|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128110": {
+      "content": "<|reserved_special_token_102|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128111": {
+      "content": "<|reserved_special_token_103|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128112": {
+      "content": "<|reserved_special_token_104|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128113": {
+      "content": "<|reserved_special_token_105|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128114": {
+      "content": "<|reserved_special_token_106|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128115": {
+      "content": "<|reserved_special_token_107|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128116": {
+      "content": "<|reserved_special_token_108|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128117": {
+      "content": "<|reserved_special_token_109|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128118": {
+      "content": "<|reserved_special_token_110|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128119": {
+      "content": "<|reserved_special_token_111|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128120": {
+      "content": "<|reserved_special_token_112|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128121": {
+      "content": "<|reserved_special_token_113|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128122": {
+      "content": "<|reserved_special_token_114|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128123": {
+      "content": "<|reserved_special_token_115|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128124": {
+      "content": "<|reserved_special_token_116|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128125": {
+      "content": "<|reserved_special_token_117|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128126": {
+      "content": "<|reserved_special_token_118|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128127": {
+      "content": "<|reserved_special_token_119|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128128": {
+      "content": "<|reserved_special_token_120|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128129": {
+      "content": "<|reserved_special_token_121|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128130": {
+      "content": "<|reserved_special_token_122|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128131": {
+      "content": "<|reserved_special_token_123|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128132": {
+      "content": "<|reserved_special_token_124|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128133": {
+      "content": "<|reserved_special_token_125|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128134": {
+      "content": "<|reserved_special_token_126|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128135": {
+      "content": "<|reserved_special_token_127|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128136": {
+      "content": "<|reserved_special_token_128|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128137": {
+      "content": "<|reserved_special_token_129|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128138": {
+      "content": "<|reserved_special_token_130|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128139": {
+      "content": "<|reserved_special_token_131|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128140": {
+      "content": "<|reserved_special_token_132|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128141": {
+      "content": "<|reserved_special_token_133|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128142": {
+      "content": "<|reserved_special_token_134|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128143": {
+      "content": "<|reserved_special_token_135|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128144": {
+      "content": "<|reserved_special_token_136|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128145": {
+      "content": "<|reserved_special_token_137|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128146": {
+      "content": "<|reserved_special_token_138|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128147": {
+      "content": "<|reserved_special_token_139|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128148": {
+      "content": "<|reserved_special_token_140|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128149": {
+      "content": "<|reserved_special_token_141|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128150": {
+      "content": "<|reserved_special_token_142|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128151": {
+      "content": "<|reserved_special_token_143|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128152": {
+      "content": "<|reserved_special_token_144|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128153": {
+      "content": "<|reserved_special_token_145|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128154": {
+      "content": "<|reserved_special_token_146|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128155": {
+      "content": "<|reserved_special_token_147|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128156": {
+      "content": "<|reserved_special_token_148|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128157": {
+      "content": "<|reserved_special_token_149|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128158": {
+      "content": "<|reserved_special_token_150|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128159": {
+      "content": "<|reserved_special_token_151|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128160": {
+      "content": "<|reserved_special_token_152|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128161": {
+      "content": "<|reserved_special_token_153|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128162": {
+      "content": "<|reserved_special_token_154|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128163": {
+      "content": "<|reserved_special_token_155|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128164": {
+      "content": "<|reserved_special_token_156|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128165": {
+      "content": "<|reserved_special_token_157|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128166": {
+      "content": "<|reserved_special_token_158|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128167": {
+      "content": "<|reserved_special_token_159|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128168": {
+      "content": "<|reserved_special_token_160|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128169": {
+      "content": "<|reserved_special_token_161|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128170": {
+      "content": "<|reserved_special_token_162|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128171": {
+      "content": "<|reserved_special_token_163|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128172": {
+      "content": "<|reserved_special_token_164|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128173": {
+      "content": "<|reserved_special_token_165|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128174": {
+      "content": "<|reserved_special_token_166|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128175": {
+      "content": "<|reserved_special_token_167|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128176": {
+      "content": "<|reserved_special_token_168|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128177": {
+      "content": "<|reserved_special_token_169|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128178": {
+      "content": "<|reserved_special_token_170|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128179": {
+      "content": "<|reserved_special_token_171|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128180": {
+      "content": "<|reserved_special_token_172|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128181": {
+      "content": "<|reserved_special_token_173|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128182": {
+      "content": "<|reserved_special_token_174|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128183": {
+      "content": "<|reserved_special_token_175|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128184": {
+      "content": "<|reserved_special_token_176|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128185": {
+      "content": "<|reserved_special_token_177|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128186": {
+      "content": "<|reserved_special_token_178|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128187": {
+      "content": "<|reserved_special_token_179|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128188": {
+      "content": "<|reserved_special_token_180|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128189": {
+      "content": "<|reserved_special_token_181|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128190": {
+      "content": "<|reserved_special_token_182|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128191": {
+      "content": "<|reserved_special_token_183|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128192": {
+      "content": "<|reserved_special_token_184|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128193": {
+      "content": "<|reserved_special_token_185|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128194": {
+      "content": "<|reserved_special_token_186|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128195": {
+      "content": "<|reserved_special_token_187|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128196": {
+      "content": "<|reserved_special_token_188|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128197": {
+      "content": "<|reserved_special_token_189|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128198": {
+      "content": "<|reserved_special_token_190|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128199": {
+      "content": "<|reserved_special_token_191|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128200": {
+      "content": "<|reserved_special_token_192|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128201": {
+      "content": "<|reserved_special_token_193|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128202": {
+      "content": "<|reserved_special_token_194|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128203": {
+      "content": "<|reserved_special_token_195|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128204": {
+      "content": "<|reserved_special_token_196|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128205": {
+      "content": "<|reserved_special_token_197|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128206": {
+      "content": "<|reserved_special_token_198|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128207": {
+      "content": "<|reserved_special_token_199|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128208": {
+      "content": "<|reserved_special_token_200|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128209": {
+      "content": "<|reserved_special_token_201|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128210": {
+      "content": "<|reserved_special_token_202|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128211": {
+      "content": "<|reserved_special_token_203|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128212": {
+      "content": "<|reserved_special_token_204|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128213": {
+      "content": "<|reserved_special_token_205|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128214": {
+      "content": "<|reserved_special_token_206|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128215": {
+      "content": "<|reserved_special_token_207|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128216": {
+      "content": "<|reserved_special_token_208|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128217": {
+      "content": "<|reserved_special_token_209|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128218": {
+      "content": "<|reserved_special_token_210|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128219": {
+      "content": "<|reserved_special_token_211|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128220": {
+      "content": "<|reserved_special_token_212|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128221": {
+      "content": "<|reserved_special_token_213|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128222": {
+      "content": "<|reserved_special_token_214|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128223": {
+      "content": "<|reserved_special_token_215|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128224": {
+      "content": "<|reserved_special_token_216|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128225": {
+      "content": "<|reserved_special_token_217|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128226": {
+      "content": "<|reserved_special_token_218|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128227": {
+      "content": "<|reserved_special_token_219|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128228": {
+      "content": "<|reserved_special_token_220|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128229": {
+      "content": "<|reserved_special_token_221|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128230": {
+      "content": "<|reserved_special_token_222|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128231": {
+      "content": "<|reserved_special_token_223|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128232": {
+      "content": "<|reserved_special_token_224|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128233": {
+      "content": "<|reserved_special_token_225|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128234": {
+      "content": "<|reserved_special_token_226|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128235": {
+      "content": "<|reserved_special_token_227|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128236": {
+      "content": "<|reserved_special_token_228|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128237": {
+      "content": "<|reserved_special_token_229|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128238": {
+      "content": "<|reserved_special_token_230|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128239": {
+      "content": "<|reserved_special_token_231|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128240": {
+      "content": "<|reserved_special_token_232|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128241": {
+      "content": "<|reserved_special_token_233|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128242": {
+      "content": "<|reserved_special_token_234|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128243": {
+      "content": "<|reserved_special_token_235|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128244": {
+      "content": "<|reserved_special_token_236|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128245": {
+      "content": "<|reserved_special_token_237|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128246": {
+      "content": "<|reserved_special_token_238|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128247": {
+      "content": "<|reserved_special_token_239|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128248": {
+      "content": "<|reserved_special_token_240|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128249": {
+      "content": "<|reserved_special_token_241|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128250": {
+      "content": "<|reserved_special_token_242|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128251": {
+      "content": "<|reserved_special_token_243|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128252": {
+      "content": "<|reserved_special_token_244|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128253": {
+      "content": "<|reserved_special_token_245|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128254": {
+      "content": "<|reserved_special_token_246|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128255": {
+      "content": "<|reserved_special_token_247|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "bos_token": "<|begin_of_text|>",
+  "clean_up_tokenization_spaces": true,
+  "eos_token": "<|eot_id|>",
+  "extra_special_tokens": {},
+  "model_input_names": [
+    "input_ids",
+    "attention_mask"
+  ],
+  "model_max_length": 131072,
+  "pad_token": "<|finetune_right_pad_id|>",
+  "padding_side": "right",
+  "tokenizer_class": "PreTrainedTokenizerFast",
+  "unk_token": null
+}

experiments/de-en-pt-zh/checkpoint-1580/trainer_state.json

@@ -0,0 +1,2246 @@
+{
+  "best_global_step": null,
+  "best_metric": null,
+  "best_model_checkpoint": null,
+  "epoch": 5.0,
+  "eval_steps": 500,
+  "global_step": 1580,
+  "is_hyper_param_search": false,
+  "is_local_process_zero": true,
+  "is_world_process_zero": true,
+  "log_history": [
+    {
+      "epoch": 0.015822784810126583,
+      "grad_norm": 6.080485820770264,
+      "learning_rate": 0.0002,
+      "loss": 2.7599,
+      "step": 5
+    },
+    {
+      "epoch": 0.03164556962025317,
+      "grad_norm": 1.6470410823822021,
+      "learning_rate": 0.00019936548223350255,
+      "loss": 0.8662,
+      "step": 10
+    },
+    {
+      "epoch": 0.04746835443037975,
+      "grad_norm": 1.656543254852295,
+      "learning_rate": 0.00019873096446700508,
+      "loss": 0.5796,
+      "step": 15
+    },
+    {
+      "epoch": 0.06329113924050633,
+      "grad_norm": 0.9287125468254089,
+      "learning_rate": 0.00019809644670050762,
+      "loss": 0.5521,
+      "step": 20
+    },
+    {
+      "epoch": 0.07911392405063292,
+      "grad_norm": 0.9500881433486938,
+      "learning_rate": 0.00019746192893401016,
+      "loss": 0.4879,
+      "step": 25
+    },
+    {
+      "epoch": 0.0949367088607595,
+      "grad_norm": 0.6434516310691833,
+      "learning_rate": 0.00019682741116751272,
+      "loss": 0.467,
+      "step": 30
+    },
+    {
+      "epoch": 0.11075949367088607,
+      "grad_norm": 0.8102421760559082,
+      "learning_rate": 0.00019619289340101523,
+      "loss": 0.4872,
+      "step": 35
+    },
+    {
+      "epoch": 0.12658227848101267,
+      "grad_norm": 0.8061529397964478,
+      "learning_rate": 0.00019555837563451777,
+      "loss": 0.4883,
+      "step": 40
+    },
+    {
+      "epoch": 0.14240506329113925,
+      "grad_norm": 0.8552656769752502,
+      "learning_rate": 0.00019492385786802033,
+      "loss": 0.4634,
+      "step": 45
+    },
+    {
+      "epoch": 0.15822784810126583,
+      "grad_norm": 0.6702130436897278,
+      "learning_rate": 0.00019428934010152284,
+      "loss": 0.4606,
+      "step": 50
+    },
+    {
+      "epoch": 0.17405063291139242,
+      "grad_norm": 0.7621438503265381,
+      "learning_rate": 0.00019365482233502538,
+      "loss": 0.5159,
+      "step": 55
+    },
+    {
+      "epoch": 0.189873417721519,
+      "grad_norm": 0.6256672143936157,
+      "learning_rate": 0.00019302030456852794,
+      "loss": 0.4818,
+      "step": 60
+    },
+    {
+      "epoch": 0.20569620253164558,
+      "grad_norm": 0.6232669353485107,
+      "learning_rate": 0.00019238578680203048,
+      "loss": 0.4926,
+      "step": 65
+    },
+    {
+      "epoch": 0.22151898734177214,
+      "grad_norm": 0.6387396454811096,
+      "learning_rate": 0.000191751269035533,
+      "loss": 0.4841,
+      "step": 70
+    },
+    {
+      "epoch": 0.23734177215189872,
+      "grad_norm": 0.6436880230903625,
+      "learning_rate": 0.00019111675126903555,
+      "loss": 0.4871,
+      "step": 75
+    },
+    {
+      "epoch": 0.25316455696202533,
+      "grad_norm": 0.7336615920066833,
+      "learning_rate": 0.0001904822335025381,
+      "loss": 0.4568,
+      "step": 80
+    },
+    {
+      "epoch": 0.2689873417721519,
+      "grad_norm": 0.7388893961906433,
+      "learning_rate": 0.0001898477157360406,
+      "loss": 0.4553,
+      "step": 85
+    },
+    {
+      "epoch": 0.2848101265822785,
+      "grad_norm": 0.6054999828338623,
+      "learning_rate": 0.00018921319796954316,
+      "loss": 0.4405,
+      "step": 90
+    },
+    {
+      "epoch": 0.30063291139240506,
+      "grad_norm": 0.7911633253097534,
+      "learning_rate": 0.0001885786802030457,
+      "loss": 0.5219,
+      "step": 95
+    },
+    {
+      "epoch": 0.31645569620253167,
+      "grad_norm": 0.5389903783798218,
+      "learning_rate": 0.00018794416243654824,
+      "loss": 0.4525,
+      "step": 100
+    },
+    {
+      "epoch": 0.3322784810126582,
+      "grad_norm": 0.7426465749740601,
+      "learning_rate": 0.00018730964467005077,
+      "loss": 0.4856,
+      "step": 105
+    },
+    {
+      "epoch": 0.34810126582278483,
+      "grad_norm": 0.7140562534332275,
+      "learning_rate": 0.0001866751269035533,
+      "loss": 0.4918,
+      "step": 110
+    },
+    {
+      "epoch": 0.3639240506329114,
+      "grad_norm": 0.624547004699707,
+      "learning_rate": 0.00018604060913705585,
+      "loss": 0.4806,
+      "step": 115
+    },
+    {
+      "epoch": 0.379746835443038,
+      "grad_norm": 0.8368127942085266,
+      "learning_rate": 0.00018540609137055838,
+      "loss": 0.4915,
+      "step": 120
+    },
+    {
+      "epoch": 0.39556962025316456,
+      "grad_norm": 0.6069784164428711,
+      "learning_rate": 0.00018477157360406092,
+      "loss": 0.485,
+      "step": 125
+    },
+    {
+      "epoch": 0.41139240506329117,
+      "grad_norm": 0.6394964456558228,
+      "learning_rate": 0.00018413705583756346,
+      "loss": 0.4815,
+      "step": 130
+    },
+    {
+      "epoch": 0.4272151898734177,
+      "grad_norm": 0.6225919127464294,
+      "learning_rate": 0.000183502538071066,
+      "loss": 0.4744,
+      "step": 135
+    },
+    {
+      "epoch": 0.4430379746835443,
+      "grad_norm": 0.7593483328819275,
+      "learning_rate": 0.00018286802030456853,
+      "loss": 0.4505,
+      "step": 140
+    },
+    {
+      "epoch": 0.4588607594936709,
+      "grad_norm": 0.5810648202896118,
+      "learning_rate": 0.00018223350253807107,
+      "loss": 0.435,
+      "step": 145
+    },
+    {
+      "epoch": 0.47468354430379744,
+      "grad_norm": 0.7975546717643738,
+      "learning_rate": 0.00018159898477157363,
+      "loss": 0.4685,
+      "step": 150
+    },
+    {
+      "epoch": 0.49050632911392406,
+      "grad_norm": 0.6529037356376648,
+      "learning_rate": 0.00018096446700507614,
+      "loss": 0.4089,
+      "step": 155
+    },
+    {
+      "epoch": 0.5063291139240507,
+      "grad_norm": 0.7088711857795715,
+      "learning_rate": 0.00018032994923857868,
+      "loss": 0.4388,
+      "step": 160
+    },
+    {
+      "epoch": 0.5221518987341772,
+      "grad_norm": 0.5964807271957397,
+      "learning_rate": 0.00017969543147208124,
+      "loss": 0.4104,
+      "step": 165
+    },
+    {
+      "epoch": 0.5379746835443038,
+      "grad_norm": 0.6564443707466125,
+      "learning_rate": 0.00017906091370558375,
+      "loss": 0.4717,
+      "step": 170
+    },
+    {
+      "epoch": 0.5537974683544303,
+      "grad_norm": 0.6912603974342346,
+      "learning_rate": 0.0001784263959390863,
+      "loss": 0.4528,
+      "step": 175
+    },
+    {
+      "epoch": 0.569620253164557,
+      "grad_norm": 0.6068799495697021,
+      "learning_rate": 0.00017779187817258885,
+      "loss": 0.4714,
+      "step": 180
+    },
+    {
+      "epoch": 0.5854430379746836,
+      "grad_norm": 0.6037777066230774,
+      "learning_rate": 0.0001771573604060914,
+      "loss": 0.5058,
+      "step": 185
+    },
+    {
+      "epoch": 0.6012658227848101,
+      "grad_norm": 0.586581826210022,
+      "learning_rate": 0.0001765228426395939,
+      "loss": 0.439,
+      "step": 190
+    },
+    {
+      "epoch": 0.6170886075949367,
+      "grad_norm": 0.6780263185501099,
+      "learning_rate": 0.00017588832487309646,
+      "loss": 0.4269,
+      "step": 195
+    },
+    {
+      "epoch": 0.6329113924050633,
+      "grad_norm": 0.7488175630569458,
+      "learning_rate": 0.000175253807106599,
+      "loss": 0.4481,
+      "step": 200
+    },
+    {
+      "epoch": 0.6487341772151899,
+      "grad_norm": 0.6138256788253784,
+      "learning_rate": 0.0001746192893401015,
+      "loss": 0.4654,
+      "step": 205
+    },
+    {
+      "epoch": 0.6645569620253164,
+      "grad_norm": 0.7444900274276733,
+      "learning_rate": 0.00017398477157360407,
+      "loss": 0.4165,
+      "step": 210
+    },
+    {
+      "epoch": 0.680379746835443,
+      "grad_norm": 0.6231881380081177,
+      "learning_rate": 0.0001733502538071066,
+      "loss": 0.4469,
+      "step": 215
+    },
+    {
+      "epoch": 0.6962025316455697,
+      "grad_norm": 0.6703267097473145,
+      "learning_rate": 0.00017271573604060915,
+      "loss": 0.4279,
+      "step": 220
+    },
+    {
+      "epoch": 0.7120253164556962,
+      "grad_norm": 0.4783850312232971,
+      "learning_rate": 0.00017208121827411168,
+      "loss": 0.4654,
+      "step": 225
+    },
+    {
+      "epoch": 0.7278481012658228,
+      "grad_norm": 0.7536376118659973,
+      "learning_rate": 0.00017144670050761422,
+      "loss": 0.4537,
+      "step": 230
+    },
+    {
+      "epoch": 0.7436708860759493,
+      "grad_norm": 0.7595057487487793,
+      "learning_rate": 0.00017081218274111676,
+      "loss": 0.486,
+      "step": 235
+    },
+    {
+      "epoch": 0.759493670886076,
+      "grad_norm": 0.8007245659828186,
+      "learning_rate": 0.0001701776649746193,
+      "loss": 0.3919,
+      "step": 240
+    },
+    {
+      "epoch": 0.7753164556962026,
+      "grad_norm": 0.6637867093086243,
+      "learning_rate": 0.00016954314720812183,
+      "loss": 0.418,
+      "step": 245
+    },
+    {
+      "epoch": 0.7911392405063291,
+      "grad_norm": 0.653685986995697,
+      "learning_rate": 0.00016890862944162437,
+      "loss": 0.4326,
+      "step": 250
+    },
+    {
+      "epoch": 0.8069620253164557,
+      "grad_norm": 0.6465121507644653,
+      "learning_rate": 0.00016827411167512693,
+      "loss": 0.445,
+      "step": 255
+    },
+    {
+      "epoch": 0.8227848101265823,
+      "grad_norm": 0.6836300492286682,
+      "learning_rate": 0.00016763959390862944,
+      "loss": 0.4348,
+      "step": 260
+    },
+    {
+      "epoch": 0.8386075949367089,
+      "grad_norm": 0.7092205882072449,
+      "learning_rate": 0.00016700507614213198,
+      "loss": 0.4081,
+      "step": 265
+    },
+    {
+      "epoch": 0.8544303797468354,
+      "grad_norm": 0.7940587997436523,
+      "learning_rate": 0.00016637055837563454,
+      "loss": 0.4316,
+      "step": 270
+    },
+    {
+      "epoch": 0.870253164556962,
+      "grad_norm": 0.6765463352203369,
+      "learning_rate": 0.00016573604060913705,
+      "loss": 0.45,
+      "step": 275
+    },
+    {
+      "epoch": 0.8860759493670886,
+      "grad_norm": 0.8914448022842407,
+      "learning_rate": 0.0001651015228426396,
+      "loss": 0.4202,
+      "step": 280
+    },
+    {
+      "epoch": 0.9018987341772152,
+      "grad_norm": 0.6309791207313538,
+      "learning_rate": 0.00016446700507614215,
+      "loss": 0.4308,
+      "step": 285
+    },
+    {
+      "epoch": 0.9177215189873418,
+      "grad_norm": 0.7343917489051819,
+      "learning_rate": 0.0001638324873096447,
+      "loss": 0.4387,
+      "step": 290
+    },
+    {
+      "epoch": 0.9335443037974683,
+      "grad_norm": 0.5776176452636719,
+      "learning_rate": 0.0001631979695431472,
+      "loss": 0.4239,
+      "step": 295
+    },
+    {
+      "epoch": 0.9493670886075949,
+      "grad_norm": 0.6677407026290894,
+      "learning_rate": 0.00016256345177664976,
+      "loss": 0.4481,
+      "step": 300
+    },
+    {
+      "epoch": 0.9651898734177216,
+      "grad_norm": 0.6818298101425171,
+      "learning_rate": 0.0001619289340101523,
+      "loss": 0.4089,
+      "step": 305
+    },
+    {
+      "epoch": 0.9810126582278481,
+      "grad_norm": 0.9606236219406128,
+      "learning_rate": 0.0001612944162436548,
+      "loss": 0.4491,
+      "step": 310
+    },
+    {
+      "epoch": 0.9968354430379747,
+      "grad_norm": 0.6414772272109985,
+      "learning_rate": 0.00016065989847715737,
+      "loss": 0.4639,
+      "step": 315
+    },
+    {
+      "epoch": 1.0126582278481013,
+      "grad_norm": 0.5724285244941711,
+      "learning_rate": 0.0001600253807106599,
+      "loss": 0.3448,
+      "step": 320
+    },
+    {
+      "epoch": 1.0284810126582278,
+      "grad_norm": 0.8723211884498596,
+      "learning_rate": 0.00015939086294416242,
+      "loss": 0.3483,
+      "step": 325
+    },
+    {
+      "epoch": 1.0443037974683544,
+      "grad_norm": 0.6645333170890808,
+      "learning_rate": 0.00015875634517766498,
+      "loss": 0.3136,
+      "step": 330
+    },
+    {
+      "epoch": 1.0601265822784811,
+      "grad_norm": 0.7657269239425659,
+      "learning_rate": 0.00015812182741116752,
+      "loss": 0.3426,
+      "step": 335
+    },
+    {
+      "epoch": 1.0759493670886076,
+      "grad_norm": 0.6551052927970886,
+      "learning_rate": 0.00015748730964467006,
+      "loss": 0.3482,
+      "step": 340
+    },
+    {
+      "epoch": 1.0917721518987342,
+      "grad_norm": 0.7337777018547058,
+      "learning_rate": 0.0001568527918781726,
+      "loss": 0.3128,
+      "step": 345
+    },
+    {
+      "epoch": 1.1075949367088607,
+      "grad_norm": 0.8943917751312256,
+      "learning_rate": 0.00015621827411167513,
+      "loss": 0.3134,
+      "step": 350
+    },
+    {
+      "epoch": 1.1234177215189873,
+      "grad_norm": 0.71893310546875,
+      "learning_rate": 0.00015558375634517767,
+      "loss": 0.2815,
+      "step": 355
+    },
+    {
+      "epoch": 1.139240506329114,
+      "grad_norm": 0.7433806657791138,
+      "learning_rate": 0.0001549492385786802,
+      "loss": 0.3216,
+      "step": 360
+    },
+    {
+      "epoch": 1.1550632911392404,
+      "grad_norm": 0.7051471471786499,
+      "learning_rate": 0.00015431472081218274,
+      "loss": 0.3719,
+      "step": 365
+    },
+    {
+      "epoch": 1.1708860759493671,
+      "grad_norm": 0.6804415583610535,
+      "learning_rate": 0.00015368020304568528,
+      "loss": 0.3449,
+      "step": 370
+    },
+    {
+      "epoch": 1.1867088607594938,
+      "grad_norm": 0.9129506945610046,
+      "learning_rate": 0.00015304568527918784,
+      "loss": 0.3574,
+      "step": 375
+    },
+    {
+      "epoch": 1.2025316455696202,
+      "grad_norm": 0.8387975692749023,
+      "learning_rate": 0.00015241116751269035,
+      "loss": 0.3321,
+      "step": 380
+    },
+    {
+      "epoch": 1.2183544303797469,
+      "grad_norm": 0.689298152923584,
+      "learning_rate": 0.0001517766497461929,
+      "loss": 0.3156,
+      "step": 385
+    },
+    {
+      "epoch": 1.2341772151898733,
+      "grad_norm": 0.681196928024292,
+      "learning_rate": 0.00015114213197969545,
+      "loss": 0.3371,
+      "step": 390
+    },
+    {
+      "epoch": 1.25,
+      "grad_norm": 0.8151982426643372,
+      "learning_rate": 0.00015050761421319796,
+      "loss": 0.3356,
+      "step": 395
+    },
+    {
+      "epoch": 1.2658227848101267,
+      "grad_norm": 0.789669394493103,
+      "learning_rate": 0.0001498730964467005,
+      "loss": 0.3063,
+      "step": 400
+    },
+    {
+      "epoch": 1.2816455696202531,
+      "grad_norm": 0.7262426614761353,
+      "learning_rate": 0.00014923857868020306,
+      "loss": 0.3179,
+      "step": 405
+    },
+    {
+      "epoch": 1.2974683544303798,
+      "grad_norm": 0.717400074005127,
+      "learning_rate": 0.0001486040609137056,
+      "loss": 0.3266,
+      "step": 410
+    },
+    {
+      "epoch": 1.3132911392405062,
+      "grad_norm": 0.8459286093711853,
+      "learning_rate": 0.0001479695431472081,
+      "loss": 0.3323,
+      "step": 415
+    },
+    {
+      "epoch": 1.3291139240506329,
+      "grad_norm": 0.831169843673706,
+      "learning_rate": 0.00014733502538071067,
+      "loss": 0.3101,
+      "step": 420
+    },
+    {
+      "epoch": 1.3449367088607596,
+      "grad_norm": 0.6811087727546692,
+      "learning_rate": 0.0001467005076142132,
+      "loss": 0.3166,
+      "step": 425
+    },
+    {
+      "epoch": 1.360759493670886,
+      "grad_norm": 0.813766360282898,
+      "learning_rate": 0.00014606598984771572,
+      "loss": 0.3274,
+      "step": 430
+    },
+    {
+      "epoch": 1.3765822784810127,
+      "grad_norm": 0.7633799314498901,
+      "learning_rate": 0.00014543147208121828,
+      "loss": 0.3351,
+      "step": 435
+    },
+    {
+      "epoch": 1.3924050632911391,
+      "grad_norm": 0.7631050944328308,
+      "learning_rate": 0.00014479695431472082,
+      "loss": 0.3045,
+      "step": 440
+    },
+    {
+      "epoch": 1.4082278481012658,
+      "grad_norm": 0.8526734709739685,
+      "learning_rate": 0.00014416243654822336,
+      "loss": 0.3479,
+      "step": 445
+    },
+    {
+      "epoch": 1.4240506329113924,
+      "grad_norm": 0.7672425508499146,
+      "learning_rate": 0.0001435279187817259,
+      "loss": 0.3302,
+      "step": 450
+    },
+    {
+      "epoch": 1.439873417721519,
+      "grad_norm": 0.8053662776947021,
+      "learning_rate": 0.00014289340101522843,
+      "loss": 0.2994,
+      "step": 455
+    },
+    {
+      "epoch": 1.4556962025316456,
+      "grad_norm": 0.9087522625923157,
+      "learning_rate": 0.00014225888324873097,
+      "loss": 0.3233,
+      "step": 460
+    },
+    {
+      "epoch": 1.4715189873417722,
+      "grad_norm": 0.7690944075584412,
+      "learning_rate": 0.0001416243654822335,
+      "loss": 0.2887,
+      "step": 465
+    },
+    {
+      "epoch": 1.4873417721518987,
+      "grad_norm": 0.8058143854141235,
+      "learning_rate": 0.00014098984771573604,
+      "loss": 0.2982,
+      "step": 470
+    },
+    {
+      "epoch": 1.5031645569620253,
+      "grad_norm": 0.7568528056144714,
+      "learning_rate": 0.00014035532994923858,
+      "loss": 0.3103,
+      "step": 475
+    },
+    {
+      "epoch": 1.518987341772152,
+      "grad_norm": 0.755690336227417,
+      "learning_rate": 0.00013972081218274114,
+      "loss": 0.3009,
+      "step": 480
+    },
+    {
+      "epoch": 1.5348101265822784,
+      "grad_norm": 0.8576112985610962,
+      "learning_rate": 0.00013908629441624365,
+      "loss": 0.3317,
+      "step": 485
+    },
+    {
+      "epoch": 1.5506329113924051,
+      "grad_norm": 0.8156176805496216,
+      "learning_rate": 0.0001384517766497462,
+      "loss": 0.3187,
+      "step": 490
+    },
+    {
+      "epoch": 1.5664556962025316,
+      "grad_norm": 0.8467602729797363,
+      "learning_rate": 0.00013781725888324875,
+      "loss": 0.3155,
+      "step": 495
+    },
+    {
+      "epoch": 1.5822784810126582,
+      "grad_norm": 0.8999772071838379,
+      "learning_rate": 0.00013718274111675126,
+      "loss": 0.3402,
+      "step": 500
+    },
+    {
+      "epoch": 1.5981012658227849,
+      "grad_norm": 0.808548092842102,
+      "learning_rate": 0.0001365482233502538,
+      "loss": 0.296,
+      "step": 505
+    },
+    {
+      "epoch": 1.6139240506329116,
+      "grad_norm": 0.7033184170722961,
+      "learning_rate": 0.00013591370558375636,
+      "loss": 0.3052,
+      "step": 510
+    },
+    {
+      "epoch": 1.629746835443038,
+      "grad_norm": 0.8576298356056213,
+      "learning_rate": 0.00013527918781725887,
+      "loss": 0.3253,
+      "step": 515
+    },
+    {
+      "epoch": 1.6455696202531644,
+      "grad_norm": 0.6722769141197205,
+      "learning_rate": 0.0001346446700507614,
+      "loss": 0.2941,
+      "step": 520
+    },
+    {
+      "epoch": 1.6613924050632911,
+      "grad_norm": 0.6628623008728027,
+      "learning_rate": 0.00013401015228426397,
+      "loss": 0.3016,
+      "step": 525
+    },
+    {
+      "epoch": 1.6772151898734178,
+      "grad_norm": 0.9596195220947266,
+      "learning_rate": 0.0001333756345177665,
+      "loss": 0.3234,
+      "step": 530
+    },
+    {
+      "epoch": 1.6930379746835444,
+      "grad_norm": 0.8452304005622864,
+      "learning_rate": 0.00013274111675126905,
+      "loss": 0.3458,
+      "step": 535
+    },
+    {
+      "epoch": 1.7088607594936709,
+      "grad_norm": 0.7414538860321045,
+      "learning_rate": 0.00013210659898477158,
+      "loss": 0.3133,
+      "step": 540
+    },
+    {
+      "epoch": 1.7246835443037973,
+      "grad_norm": 0.6833527088165283,
+      "learning_rate": 0.00013147208121827412,
+      "loss": 0.3008,
+      "step": 545
+    },
+    {
+      "epoch": 1.740506329113924,
+      "grad_norm": 0.8556643724441528,
+      "learning_rate": 0.00013083756345177666,
+      "loss": 0.3035,
+      "step": 550
+    },
+    {
+      "epoch": 1.7563291139240507,
+      "grad_norm": 0.9071274399757385,
+      "learning_rate": 0.0001302030456852792,
+      "loss": 0.304,
+      "step": 555
+    },
+    {
+      "epoch": 1.7721518987341773,
+      "grad_norm": 1.0122044086456299,
+      "learning_rate": 0.00012956852791878173,
+      "loss": 0.2956,
+      "step": 560
+    },
+    {
+      "epoch": 1.7879746835443038,
+      "grad_norm": 0.9109578132629395,
+      "learning_rate": 0.00012893401015228427,
+      "loss": 0.3101,
+      "step": 565
+    },
+    {
+      "epoch": 1.8037974683544302,
+      "grad_norm": 0.9648177623748779,
+      "learning_rate": 0.0001282994923857868,
+      "loss": 0.2987,
+      "step": 570
+    },
+    {
+      "epoch": 1.8196202531645569,
+      "grad_norm": 0.8060092926025391,
+      "learning_rate": 0.00012766497461928934,
+      "loss": 0.3011,
+      "step": 575
+    },
+    {
+      "epoch": 1.8354430379746836,
+      "grad_norm": 0.7650238871574402,
+      "learning_rate": 0.00012703045685279188,
+      "loss": 0.3207,
+      "step": 580
+    },
+    {
+      "epoch": 1.8512658227848102,
+      "grad_norm": 1.0391228199005127,
+      "learning_rate": 0.00012639593908629442,
+      "loss": 0.3,
+      "step": 585
+    },
+    {
+      "epoch": 1.8670886075949367,
+      "grad_norm": 0.7758641242980957,
+      "learning_rate": 0.00012576142131979695,
+      "loss": 0.3109,
+      "step": 590
+    },
+    {
+      "epoch": 1.8829113924050633,
+      "grad_norm": 0.819521427154541,
+      "learning_rate": 0.0001251269035532995,
+      "loss": 0.2921,
+      "step": 595
+    },
+    {
+      "epoch": 1.8987341772151898,
+      "grad_norm": 0.7439246773719788,
+      "learning_rate": 0.00012449238578680205,
+      "loss": 0.3091,
+      "step": 600
+    },
+    {
+      "epoch": 1.9145569620253164,
+      "grad_norm": 0.8022212386131287,
+      "learning_rate": 0.00012385786802030456,
+      "loss": 0.3252,
+      "step": 605
+    },
+    {
+      "epoch": 1.9303797468354431,
+      "grad_norm": 1.0216495990753174,
+      "learning_rate": 0.0001232233502538071,
+      "loss": 0.3434,
+      "step": 610
+    },
+    {
+      "epoch": 1.9462025316455698,
+      "grad_norm": 0.821251630783081,
+      "learning_rate": 0.00012258883248730966,
+      "loss": 0.325,
+      "step": 615
+    },
+    {
+      "epoch": 1.9620253164556962,
+      "grad_norm": 0.744149386882782,
+      "learning_rate": 0.00012195431472081217,
+      "loss": 0.3286,
+      "step": 620
+    },
+    {
+      "epoch": 1.9778481012658227,
+      "grad_norm": 0.8488931655883789,
+      "learning_rate": 0.00012131979695431472,
+      "loss": 0.3048,
+      "step": 625
+    },
+    {
+      "epoch": 1.9936708860759493,
+      "grad_norm": 0.8845570087432861,
+      "learning_rate": 0.00012068527918781727,
+      "loss": 0.3027,
+      "step": 630
+    },
+    {
+      "epoch": 2.009493670886076,
+      "grad_norm": 0.6925209164619446,
+      "learning_rate": 0.00012005076142131981,
+      "loss": 0.2111,
+      "step": 635
+    },
+    {
+      "epoch": 2.0253164556962027,
+      "grad_norm": 1.101040244102478,
+      "learning_rate": 0.00011941624365482233,
+      "loss": 0.1921,
+      "step": 640
+    },
+    {
+      "epoch": 2.041139240506329,
+      "grad_norm": 1.030375361442566,
+      "learning_rate": 0.00011878172588832489,
+      "loss": 0.1878,
+      "step": 645
+    },
+    {
+      "epoch": 2.0569620253164556,
+      "grad_norm": 0.8349786996841431,
+      "learning_rate": 0.00011814720812182742,
+      "loss": 0.1839,
+      "step": 650
+    },
+    {
+      "epoch": 2.0727848101265822,
+      "grad_norm": 0.7567915320396423,
+      "learning_rate": 0.00011751269035532995,
+      "loss": 0.1978,
+      "step": 655
+    },
+    {
+      "epoch": 2.088607594936709,
+      "grad_norm": 0.9065817594528198,
+      "learning_rate": 0.0001168781725888325,
+      "loss": 0.1904,
+      "step": 660
+    },
+    {
+      "epoch": 2.1044303797468356,
+      "grad_norm": 0.8686302900314331,
+      "learning_rate": 0.00011624365482233503,
+      "loss": 0.1898,
+      "step": 665
+    },
+    {
+      "epoch": 2.1202531645569622,
+      "grad_norm": 0.81605064868927,
+      "learning_rate": 0.00011560913705583758,
+      "loss": 0.186,
+      "step": 670
+    },
+    {
+      "epoch": 2.1360759493670884,
+      "grad_norm": 0.8041939735412598,
+      "learning_rate": 0.0001149746192893401,
+      "loss": 0.1988,
+      "step": 675
+    },
+    {
+      "epoch": 2.151898734177215,
+      "grad_norm": 0.9789828658103943,
+      "learning_rate": 0.00011434010152284264,
+      "loss": 0.1886,
+      "step": 680
+    },
+    {
+      "epoch": 2.1677215189873418,
+      "grad_norm": 0.822130560874939,
+      "learning_rate": 0.0001137055837563452,
+      "loss": 0.1903,
+      "step": 685
+    },
+    {
+      "epoch": 2.1835443037974684,
+      "grad_norm": 0.8661246299743652,
+      "learning_rate": 0.00011307106598984772,
+      "loss": 0.1911,
+      "step": 690
+    },
+    {
+      "epoch": 2.199367088607595,
+      "grad_norm": 0.814487636089325,
+      "learning_rate": 0.00011243654822335025,
+      "loss": 0.1801,
+      "step": 695
+    },
+    {
+      "epoch": 2.2151898734177213,
+      "grad_norm": 0.9888129830360413,
+      "learning_rate": 0.0001118020304568528,
+      "loss": 0.1859,
+      "step": 700
+    },
+    {
+      "epoch": 2.231012658227848,
+      "grad_norm": 0.790763258934021,
+      "learning_rate": 0.00011116751269035533,
+      "loss": 0.1834,
+      "step": 705
+    },
+    {
+      "epoch": 2.2468354430379747,
+      "grad_norm": 0.7045260667800903,
+      "learning_rate": 0.00011053299492385786,
+      "loss": 0.1769,
+      "step": 710
+    },
+    {
+      "epoch": 2.2626582278481013,
+      "grad_norm": 0.8450356125831604,
+      "learning_rate": 0.00010989847715736041,
+      "loss": 0.1912,
+      "step": 715
+    },
+    {
+      "epoch": 2.278481012658228,
+      "grad_norm": 0.9821575284004211,
+      "learning_rate": 0.00010926395939086295,
+      "loss": 0.194,
+      "step": 720
+    },
+    {
+      "epoch": 2.2943037974683547,
+      "grad_norm": 0.827716588973999,
+      "learning_rate": 0.00010862944162436547,
+      "loss": 0.1952,
+      "step": 725
+    },
+    {
+      "epoch": 2.310126582278481,
+      "grad_norm": 1.0735716819763184,
+      "learning_rate": 0.00010799492385786802,
+      "loss": 0.1962,
+      "step": 730
+    },
+    {
+      "epoch": 2.3259493670886076,
+      "grad_norm": 0.759575605392456,
+      "learning_rate": 0.00010736040609137057,
+      "loss": 0.1899,
+      "step": 735
+    },
+    {
+      "epoch": 2.3417721518987342,
+      "grad_norm": 1.0905171632766724,
+      "learning_rate": 0.00010672588832487308,
+      "loss": 0.1836,
+      "step": 740
+    },
+    {
+      "epoch": 2.357594936708861,
+      "grad_norm": 0.9864721894264221,
+      "learning_rate": 0.00010609137055837564,
+      "loss": 0.1988,
+      "step": 745
+    },
+    {
+      "epoch": 2.3734177215189876,
+      "grad_norm": 0.8583397269248962,
+      "learning_rate": 0.00010545685279187819,
+      "loss": 0.2032,
+      "step": 750
+    },
+    {
+      "epoch": 2.3892405063291138,
+      "grad_norm": 0.8750898241996765,
+      "learning_rate": 0.00010482233502538072,
+      "loss": 0.1913,
+      "step": 755
+    },
+    {
+      "epoch": 2.4050632911392404,
+      "grad_norm": 0.8484179377555847,
+      "learning_rate": 0.00010418781725888325,
+      "loss": 0.1932,
+      "step": 760
+    },
+    {
+      "epoch": 2.420886075949367,
+      "grad_norm": 0.8363800048828125,
+      "learning_rate": 0.0001035532994923858,
+      "loss": 0.1961,
+      "step": 765
+    },
+    {
+      "epoch": 2.4367088607594938,
+      "grad_norm": 1.0165894031524658,
+      "learning_rate": 0.00010291878172588833,
+      "loss": 0.1997,
+      "step": 770
+    },
+    {
+      "epoch": 2.4525316455696204,
+      "grad_norm": 0.9139420986175537,
+      "learning_rate": 0.00010228426395939086,
+      "loss": 0.185,
+      "step": 775
+    },
+    {
+      "epoch": 2.4683544303797467,
+      "grad_norm": 0.8798001408576965,
+      "learning_rate": 0.0001016497461928934,
+      "loss": 0.1842,
+      "step": 780
+    },
+    {
+      "epoch": 2.4841772151898733,
+      "grad_norm": 0.8399237394332886,
+      "learning_rate": 0.00010101522842639594,
+      "loss": 0.1759,
+      "step": 785
+    },
+    {
+      "epoch": 2.5,
+      "grad_norm": 0.883194625377655,
+      "learning_rate": 0.0001003807106598985,
+      "loss": 0.2027,
+      "step": 790
+    },
+    {
+      "epoch": 2.5158227848101267,
+      "grad_norm": 1.1090173721313477,
+      "learning_rate": 9.974619289340103e-05,
+      "loss": 0.192,
+      "step": 795
+    },
+    {
+      "epoch": 2.5316455696202533,
+      "grad_norm": 0.8255505561828613,
+      "learning_rate": 9.911167512690355e-05,
+      "loss": 0.1962,
+      "step": 800
+    },
+    {
+      "epoch": 2.5474683544303796,
+      "grad_norm": 1.0160300731658936,
+      "learning_rate": 9.847715736040609e-05,
+      "loss": 0.1932,
+      "step": 805
+    },
+    {
+      "epoch": 2.5632911392405062,
+      "grad_norm": 0.7054086327552795,
+      "learning_rate": 9.784263959390864e-05,
+      "loss": 0.186,
+      "step": 810
+    },
+    {
+      "epoch": 2.579113924050633,
+      "grad_norm": 0.7671273946762085,
+      "learning_rate": 9.720812182741116e-05,
+      "loss": 0.1851,
+      "step": 815
+    },
+    {
+      "epoch": 2.5949367088607596,
+      "grad_norm": 0.9428186416625977,
+      "learning_rate": 9.657360406091371e-05,
+      "loss": 0.1903,
+      "step": 820
+    },
+    {
+      "epoch": 2.6107594936708862,
+      "grad_norm": 0.8303772807121277,
+      "learning_rate": 9.593908629441625e-05,
+      "loss": 0.1877,
+      "step": 825
+    },
+    {
+      "epoch": 2.6265822784810124,
+      "grad_norm": 0.9931467175483704,
+      "learning_rate": 9.530456852791879e-05,
+      "loss": 0.2038,
+      "step": 830
+    },
+    {
+      "epoch": 2.642405063291139,
+      "grad_norm": 0.8338640928268433,
+      "learning_rate": 9.467005076142132e-05,
+      "loss": 0.1782,
+      "step": 835
+    },
+    {
+      "epoch": 2.6582278481012658,
+      "grad_norm": 1.0357074737548828,
+      "learning_rate": 9.403553299492386e-05,
+      "loss": 0.204,
+      "step": 840
+    },
+    {
+      "epoch": 2.6740506329113924,
+      "grad_norm": 0.7227370142936707,
+      "learning_rate": 9.34010152284264e-05,
+      "loss": 0.1871,
+      "step": 845
+    },
+    {
+      "epoch": 2.689873417721519,
+      "grad_norm": 0.8483996987342834,
+      "learning_rate": 9.276649746192894e-05,
+      "loss": 0.1884,
+      "step": 850
+    },
+    {
+      "epoch": 2.7056962025316453,
+      "grad_norm": 0.7692596912384033,
+      "learning_rate": 9.213197969543149e-05,
+      "loss": 0.1987,
+      "step": 855
+    },
+    {
+      "epoch": 2.721518987341772,
+      "grad_norm": 0.9548447728157043,
+      "learning_rate": 9.149746192893401e-05,
+      "loss": 0.1836,
+      "step": 860
+    },
+    {
+      "epoch": 2.7373417721518987,
+      "grad_norm": 0.9363961219787598,
+      "learning_rate": 9.086294416243655e-05,
+      "loss": 0.1815,
+      "step": 865
+    },
+    {
+      "epoch": 2.7531645569620253,
+      "grad_norm": 1.0330653190612793,
+      "learning_rate": 9.02284263959391e-05,
+      "loss": 0.1957,
+      "step": 870
+    },
+    {
+      "epoch": 2.768987341772152,
+      "grad_norm": 0.8239451050758362,
+      "learning_rate": 8.959390862944162e-05,
+      "loss": 0.1945,
+      "step": 875
+    },
+    {
+      "epoch": 2.7848101265822782,
+      "grad_norm": 0.7226342558860779,
+      "learning_rate": 8.895939086294417e-05,
+      "loss": 0.1819,
+      "step": 880
+    },
+    {
+      "epoch": 2.8006329113924053,
+      "grad_norm": 1.2467807531356812,
+      "learning_rate": 8.83248730964467e-05,
+      "loss": 0.1723,
+      "step": 885
+    },
+    {
+      "epoch": 2.8164556962025316,
+      "grad_norm": 0.8140600919723511,
+      "learning_rate": 8.769035532994924e-05,
+      "loss": 0.1898,
+      "step": 890
+    },
+    {
+      "epoch": 2.8322784810126582,
+      "grad_norm": 0.8722625970840454,
+      "learning_rate": 8.705583756345178e-05,
+      "loss": 0.1895,
+      "step": 895
+    },
+    {
+      "epoch": 2.848101265822785,
+      "grad_norm": 1.1079012155532837,
+      "learning_rate": 8.642131979695432e-05,
+      "loss": 0.1999,
+      "step": 900
+    },
+    {
+      "epoch": 2.8639240506329116,
+      "grad_norm": 0.8730645775794983,
+      "learning_rate": 8.578680203045685e-05,
+      "loss": 0.1891,
+      "step": 905
+    },
+    {
+      "epoch": 2.879746835443038,
+      "grad_norm": 0.9077411890029907,
+      "learning_rate": 8.515228426395939e-05,
+      "loss": 0.1756,
+      "step": 910
+    },
+    {
+      "epoch": 2.8955696202531644,
+      "grad_norm": 0.9811768531799316,
+      "learning_rate": 8.451776649746194e-05,
+      "loss": 0.1937,
+      "step": 915
+    },
+    {
+      "epoch": 2.911392405063291,
+      "grad_norm": 0.8190383911132812,
+      "learning_rate": 8.388324873096446e-05,
+      "loss": 0.1947,
+      "step": 920
+    },
+    {
+      "epoch": 2.9272151898734178,
+      "grad_norm": 1.0100188255310059,
+      "learning_rate": 8.324873096446701e-05,
+      "loss": 0.2103,
+      "step": 925
+    },
+    {
+      "epoch": 2.9430379746835444,
+      "grad_norm": 0.7750146389007568,
+      "learning_rate": 8.261421319796955e-05,
+      "loss": 0.1828,
+      "step": 930
+    },
+    {
+      "epoch": 2.958860759493671,
+      "grad_norm": 0.7670167684555054,
+      "learning_rate": 8.197969543147207e-05,
+      "loss": 0.1848,
+      "step": 935
+    },
+    {
+      "epoch": 2.9746835443037973,
+      "grad_norm": 0.7348405718803406,
+      "learning_rate": 8.134517766497463e-05,
+      "loss": 0.1852,
+      "step": 940
+    },
+    {
+      "epoch": 2.990506329113924,
+      "grad_norm": 0.8744247555732727,
+      "learning_rate": 8.071065989847716e-05,
+      "loss": 0.1878,
+      "step": 945
+    },
+    {
+      "epoch": 3.0063291139240507,
+      "grad_norm": 0.5043367743492126,
+      "learning_rate": 8.00761421319797e-05,
+      "loss": 0.153,
+      "step": 950
+    },
+    {
+      "epoch": 3.0221518987341773,
+      "grad_norm": 0.5560520887374878,
+      "learning_rate": 7.944162436548224e-05,
+      "loss": 0.1149,
+      "step": 955
+    },
+    {
+      "epoch": 3.037974683544304,
+      "grad_norm": 0.8911687135696411,
+      "learning_rate": 7.880710659898477e-05,
+      "loss": 0.1172,
+      "step": 960
+    },
+    {
+      "epoch": 3.0537974683544302,
+      "grad_norm": 0.7934268116950989,
+      "learning_rate": 7.817258883248731e-05,
+      "loss": 0.1155,
+      "step": 965
+    },
+    {
+      "epoch": 3.069620253164557,
+      "grad_norm": 0.6288787722587585,
+      "learning_rate": 7.753807106598985e-05,
+      "loss": 0.1079,
+      "step": 970
+    },
+    {
+      "epoch": 3.0854430379746836,
+      "grad_norm": 0.6213780641555786,
+      "learning_rate": 7.69035532994924e-05,
+      "loss": 0.1196,
+      "step": 975
+    },
+    {
+      "epoch": 3.1012658227848102,
+      "grad_norm": 0.8186531066894531,
+      "learning_rate": 7.626903553299492e-05,
+      "loss": 0.1215,
+      "step": 980
+    },
+    {
+      "epoch": 3.117088607594937,
+      "grad_norm": 0.6212114691734314,
+      "learning_rate": 7.563451776649747e-05,
+      "loss": 0.1222,
+      "step": 985
+    },
+    {
+      "epoch": 3.132911392405063,
+      "grad_norm": 0.6861391067504883,
+      "learning_rate": 7.500000000000001e-05,
+      "loss": 0.1142,
+      "step": 990
+    },
+    {
+      "epoch": 3.1487341772151898,
+      "grad_norm": 0.5754695534706116,
+      "learning_rate": 7.436548223350253e-05,
+      "loss": 0.1144,
+      "step": 995
+    },
+    {
+      "epoch": 3.1645569620253164,
+      "grad_norm": 0.9572857022285461,
+      "learning_rate": 7.373096446700508e-05,
+      "loss": 0.1157,
+      "step": 1000
+    },
+    {
+      "epoch": 3.180379746835443,
+      "grad_norm": 0.6589420437812805,
+      "learning_rate": 7.309644670050762e-05,
+      "loss": 0.1195,
+      "step": 1005
+    },
+    {
+      "epoch": 3.1962025316455698,
+      "grad_norm": 0.7942759990692139,
+      "learning_rate": 7.246192893401015e-05,
+      "loss": 0.1199,
+      "step": 1010
+    },
+    {
+      "epoch": 3.212025316455696,
+      "grad_norm": 0.7023386359214783,
+      "learning_rate": 7.182741116751269e-05,
+      "loss": 0.1198,
+      "step": 1015
+    },
+    {
+      "epoch": 3.2278481012658227,
+      "grad_norm": 0.5077189207077026,
+      "learning_rate": 7.119289340101524e-05,
+      "loss": 0.1193,
+      "step": 1020
+    },
+    {
+      "epoch": 3.2436708860759493,
+      "grad_norm": 1.0440549850463867,
+      "learning_rate": 7.055837563451776e-05,
+      "loss": 0.1141,
+      "step": 1025
+    },
+    {
+      "epoch": 3.259493670886076,
+      "grad_norm": 0.6728818416595459,
+      "learning_rate": 6.99238578680203e-05,
+      "loss": 0.1184,
+      "step": 1030
+    },
+    {
+      "epoch": 3.2753164556962027,
+      "grad_norm": 0.6550622582435608,
+      "learning_rate": 6.928934010152285e-05,
+      "loss": 0.1207,
+      "step": 1035
+    },
+    {
+      "epoch": 3.291139240506329,
+      "grad_norm": 0.9485641121864319,
+      "learning_rate": 6.865482233502538e-05,
+      "loss": 0.1185,
+      "step": 1040
+    },
+    {
+      "epoch": 3.3069620253164556,
+      "grad_norm": 0.7015439867973328,
+      "learning_rate": 6.802030456852793e-05,
+      "loss": 0.1166,
+      "step": 1045
+    },
+    {
+      "epoch": 3.3227848101265822,
+      "grad_norm": 0.7128148674964905,
+      "learning_rate": 6.738578680203046e-05,
+      "loss": 0.1211,
+      "step": 1050
+    },
+    {
+      "epoch": 3.338607594936709,
+      "grad_norm": 0.8616880178451538,
+      "learning_rate": 6.6751269035533e-05,
+      "loss": 0.1224,
+      "step": 1055
+    },
+    {
+      "epoch": 3.3544303797468356,
+      "grad_norm": 1.673384189605713,
+      "learning_rate": 6.611675126903554e-05,
+      "loss": 0.1125,
+      "step": 1060
+    },
+    {
+      "epoch": 3.370253164556962,
+      "grad_norm": 0.7952456474304199,
+      "learning_rate": 6.548223350253807e-05,
+      "loss": 0.1196,
+      "step": 1065
+    },
+    {
+      "epoch": 3.3860759493670884,
+      "grad_norm": 0.7431464791297913,
+      "learning_rate": 6.484771573604061e-05,
+      "loss": 0.1172,
+      "step": 1070
+    },
+    {
+      "epoch": 3.401898734177215,
+      "grad_norm": 0.9536241292953491,
+      "learning_rate": 6.421319796954315e-05,
+      "loss": 0.1227,
+      "step": 1075
+    },
+    {
+      "epoch": 3.4177215189873418,
+      "grad_norm": 0.6027010679244995,
+      "learning_rate": 6.35786802030457e-05,
+      "loss": 0.1183,
+      "step": 1080
+    },
+    {
+      "epoch": 3.4335443037974684,
+      "grad_norm": 0.6765082478523254,
+      "learning_rate": 6.294416243654822e-05,
+      "loss": 0.1222,
+      "step": 1085
+    },
+    {
+      "epoch": 3.449367088607595,
+      "grad_norm": 0.8334203958511353,
+      "learning_rate": 6.230964467005076e-05,
+      "loss": 0.1188,
+      "step": 1090
+    },
+    {
+      "epoch": 3.4651898734177213,
+      "grad_norm": 0.7078428864479065,
+      "learning_rate": 6.167512690355331e-05,
+      "loss": 0.1237,
+      "step": 1095
+    },
+    {
+      "epoch": 3.481012658227848,
+      "grad_norm": 0.5793551206588745,
+      "learning_rate": 6.104060913705584e-05,
+      "loss": 0.1171,
+      "step": 1100
+    },
+    {
+      "epoch": 3.4968354430379747,
+      "grad_norm": 0.816104531288147,
+      "learning_rate": 6.040609137055838e-05,
+      "loss": 0.1216,
+      "step": 1105
+    },
+    {
+      "epoch": 3.5126582278481013,
+      "grad_norm": 0.5165361166000366,
+      "learning_rate": 5.977157360406092e-05,
+      "loss": 0.123,
+      "step": 1110
+    },
+    {
+      "epoch": 3.528481012658228,
+      "grad_norm": 0.7187574505805969,
+      "learning_rate": 5.913705583756346e-05,
+      "loss": 0.1196,
+      "step": 1115
+    },
+    {
+      "epoch": 3.5443037974683547,
+      "grad_norm": 0.8947055339813232,
+      "learning_rate": 5.850253807106599e-05,
+      "loss": 0.121,
+      "step": 1120
+    },
+    {
+      "epoch": 3.560126582278481,
+      "grad_norm": 0.6461635828018188,
+      "learning_rate": 5.786802030456853e-05,
+      "loss": 0.1171,
+      "step": 1125
+    },
+    {
+      "epoch": 3.5759493670886076,
+      "grad_norm": 0.5806897282600403,
+      "learning_rate": 5.723350253807107e-05,
+      "loss": 0.1095,
+      "step": 1130
+    },
+    {
+      "epoch": 3.5917721518987342,
+      "grad_norm": 0.48626071214675903,
+      "learning_rate": 5.65989847715736e-05,
+      "loss": 0.117,
+      "step": 1135
+    },
+    {
+      "epoch": 3.607594936708861,
+      "grad_norm": 0.7649936079978943,
+      "learning_rate": 5.5964467005076145e-05,
+      "loss": 0.1214,
+      "step": 1140
+    },
+    {
+      "epoch": 3.6234177215189876,
+      "grad_norm": 0.8777173757553101,
+      "learning_rate": 5.532994923857868e-05,
+      "loss": 0.126,
+      "step": 1145
+    },
+    {
+      "epoch": 3.6392405063291138,
+      "grad_norm": 0.7628064155578613,
+      "learning_rate": 5.469543147208121e-05,
+      "loss": 0.1194,
+      "step": 1150
+    },
+    {
+      "epoch": 3.6550632911392404,
+      "grad_norm": 0.6170560717582703,
+      "learning_rate": 5.406091370558376e-05,
+      "loss": 0.119,
+      "step": 1155
+    },
+    {
+      "epoch": 3.670886075949367,
+      "grad_norm": 0.6344670057296753,
+      "learning_rate": 5.342639593908629e-05,
+      "loss": 0.1153,
+      "step": 1160
+    },
+    {
+      "epoch": 3.6867088607594938,
+      "grad_norm": 0.6660630702972412,
+      "learning_rate": 5.2791878172588836e-05,
+      "loss": 0.1224,
+      "step": 1165
+    },
+    {
+      "epoch": 3.7025316455696204,
+      "grad_norm": 0.7865625619888306,
+      "learning_rate": 5.215736040609137e-05,
+      "loss": 0.1257,
+      "step": 1170
+    },
+    {
+      "epoch": 3.7183544303797467,
+      "grad_norm": 0.611803412437439,
+      "learning_rate": 5.152284263959392e-05,
+      "loss": 0.1195,
+      "step": 1175
+    },
+    {
+      "epoch": 3.7341772151898733,
+      "grad_norm": 0.9372175931930542,
+      "learning_rate": 5.088832487309645e-05,
+      "loss": 0.1239,
+      "step": 1180
+    },
+    {
+      "epoch": 3.75,
+      "grad_norm": 0.6816895008087158,
+      "learning_rate": 5.0253807106598984e-05,
+      "loss": 0.1228,
+      "step": 1185
+    },
+    {
+      "epoch": 3.7658227848101267,
+      "grad_norm": 0.5107612013816833,
+      "learning_rate": 4.961928934010153e-05,
+      "loss": 0.1171,
+      "step": 1190
+    },
+    {
+      "epoch": 3.7816455696202533,
+      "grad_norm": 0.5778011679649353,
+      "learning_rate": 4.8984771573604064e-05,
+      "loss": 0.1253,
+      "step": 1195
+    },
+    {
+      "epoch": 3.7974683544303796,
+      "grad_norm": 0.7186105251312256,
+      "learning_rate": 4.83502538071066e-05,
+      "loss": 0.1155,
+      "step": 1200
+    },
+    {
+      "epoch": 3.8132911392405062,
+      "grad_norm": 0.7445695996284485,
+      "learning_rate": 4.771573604060914e-05,
+      "loss": 0.1176,
+      "step": 1205
+    },
+    {
+      "epoch": 3.829113924050633,
+      "grad_norm": 0.593574047088623,
+      "learning_rate": 4.7081218274111674e-05,
+      "loss": 0.1158,
+      "step": 1210
+    },
+    {
+      "epoch": 3.8449367088607596,
+      "grad_norm": 0.5334892272949219,
+      "learning_rate": 4.644670050761422e-05,
+      "loss": 0.1172,
+      "step": 1215
+    },
+    {
+      "epoch": 3.8607594936708862,
+      "grad_norm": 0.5709195137023926,
+      "learning_rate": 4.5812182741116755e-05,
+      "loss": 0.1158,
+      "step": 1220
+    },
+    {
+      "epoch": 3.8765822784810124,
+      "grad_norm": 0.6969111561775208,
+      "learning_rate": 4.517766497461929e-05,
+      "loss": 0.1198,
+      "step": 1225
+    },
+    {
+      "epoch": 3.892405063291139,
+      "grad_norm": 0.5967200398445129,
+      "learning_rate": 4.454314720812183e-05,
+      "loss": 0.115,
+      "step": 1230
+    },
+    {
+      "epoch": 3.9082278481012658,
+      "grad_norm": 0.6755451560020447,
+      "learning_rate": 4.3908629441624365e-05,
+      "loss": 0.1171,
+      "step": 1235
+    },
+    {
+      "epoch": 3.9240506329113924,
+      "grad_norm": 0.7690590620040894,
+      "learning_rate": 4.32741116751269e-05,
+      "loss": 0.117,
+      "step": 1240
+    },
+    {
+      "epoch": 3.939873417721519,
+      "grad_norm": 0.7448693513870239,
+      "learning_rate": 4.2639593908629446e-05,
+      "loss": 0.1183,
+      "step": 1245
+    },
+    {
+      "epoch": 3.9556962025316453,
+      "grad_norm": 0.6257800459861755,
+      "learning_rate": 4.200507614213198e-05,
+      "loss": 0.1122,
+      "step": 1250
+    },
+    {
+      "epoch": 3.971518987341772,
+      "grad_norm": 0.7136058807373047,
+      "learning_rate": 4.137055837563452e-05,
+      "loss": 0.1172,
+      "step": 1255
+    },
+    {
+      "epoch": 3.9873417721518987,
+      "grad_norm": 0.8732256293296814,
+      "learning_rate": 4.073604060913706e-05,
+      "loss": 0.1198,
+      "step": 1260
+    },
+    {
+      "epoch": 4.003164556962025,
+      "grad_norm": 0.3908518850803375,
+      "learning_rate": 4.010152284263959e-05,
+      "loss": 0.1182,
+      "step": 1265
+    },
+    {
+      "epoch": 4.018987341772152,
+      "grad_norm": 0.3654375672340393,
+      "learning_rate": 3.946700507614213e-05,
+      "loss": 0.0943,
+      "step": 1270
+    },
+    {
+      "epoch": 4.034810126582278,
+      "grad_norm": 0.44177666306495667,
+      "learning_rate": 3.8832487309644673e-05,
+      "loss": 0.093,
+      "step": 1275
+    },
+    {
+      "epoch": 4.050632911392405,
+      "grad_norm": 0.548538863658905,
+      "learning_rate": 3.819796954314721e-05,
+      "loss": 0.0905,
+      "step": 1280
+    },
+    {
+      "epoch": 4.0664556962025316,
+      "grad_norm": 0.34858939051628113,
+      "learning_rate": 3.756345177664975e-05,
+      "loss": 0.0882,
+      "step": 1285
+    },
+    {
+      "epoch": 4.082278481012658,
+      "grad_norm": 0.44596415758132935,
+      "learning_rate": 3.692893401015229e-05,
+      "loss": 0.091,
+      "step": 1290
+    },
+    {
+      "epoch": 4.098101265822785,
+      "grad_norm": 0.4271854758262634,
+      "learning_rate": 3.629441624365482e-05,
+      "loss": 0.0907,
+      "step": 1295
+    },
+    {
+      "epoch": 4.113924050632911,
+      "grad_norm": 0.7120389938354492,
+      "learning_rate": 3.565989847715736e-05,
+      "loss": 0.0931,
+      "step": 1300
+    },
+    {
+      "epoch": 4.129746835443038,
+      "grad_norm": 0.44612565636634827,
+      "learning_rate": 3.50253807106599e-05,
+      "loss": 0.0887,
+      "step": 1305
+    },
+    {
+      "epoch": 4.1455696202531644,
+      "grad_norm": 0.34034597873687744,
+      "learning_rate": 3.439086294416244e-05,
+      "loss": 0.0903,
+      "step": 1310
+    },
+    {
+      "epoch": 4.1613924050632916,
+      "grad_norm": 0.4155871570110321,
+      "learning_rate": 3.3756345177664975e-05,
+      "loss": 0.092,
+      "step": 1315
+    },
+    {
+      "epoch": 4.177215189873418,
+      "grad_norm": 0.5939737558364868,
+      "learning_rate": 3.312182741116752e-05,
+      "loss": 0.0928,
+      "step": 1320
+    },
+    {
+      "epoch": 4.193037974683544,
+      "grad_norm": 0.611441969871521,
+      "learning_rate": 3.248730964467005e-05,
+      "loss": 0.0897,
+      "step": 1325
+    },
+    {
+      "epoch": 4.208860759493671,
+      "grad_norm": 0.6120615601539612,
+      "learning_rate": 3.185279187817259e-05,
+      "loss": 0.0936,
+      "step": 1330
+    },
+    {
+      "epoch": 4.224683544303797,
+      "grad_norm": 0.4837358593940735,
+      "learning_rate": 3.121827411167513e-05,
+      "loss": 0.0896,
+      "step": 1335
+    },
+    {
+      "epoch": 4.2405063291139244,
+      "grad_norm": 0.40970176458358765,
+      "learning_rate": 3.0583756345177666e-05,
+      "loss": 0.0928,
+      "step": 1340
+    },
+    {
+      "epoch": 4.256329113924051,
+      "grad_norm": 0.4805232286453247,
+      "learning_rate": 2.9949238578680206e-05,
+      "loss": 0.0903,
+      "step": 1345
+    },
+    {
+      "epoch": 4.272151898734177,
+      "grad_norm": 0.4605880379676819,
+      "learning_rate": 2.9314720812182743e-05,
+      "loss": 0.0903,
+      "step": 1350
+    },
+    {
+      "epoch": 4.287974683544304,
+      "grad_norm": 0.4408807158470154,
+      "learning_rate": 2.8680203045685283e-05,
+      "loss": 0.0916,
+      "step": 1355
+    },
+    {
+      "epoch": 4.30379746835443,
+      "grad_norm": 0.4075123071670532,
+      "learning_rate": 2.8045685279187816e-05,
+      "loss": 0.0878,
+      "step": 1360
+    },
+    {
+      "epoch": 4.319620253164557,
+      "grad_norm": 0.5319828987121582,
+      "learning_rate": 2.7411167512690357e-05,
+      "loss": 0.0907,
+      "step": 1365
+    },
+    {
+      "epoch": 4.3354430379746836,
+      "grad_norm": 0.5517973899841309,
+      "learning_rate": 2.6776649746192893e-05,
+      "loss": 0.0895,
+      "step": 1370
+    },
+    {
+      "epoch": 4.35126582278481,
+      "grad_norm": 0.48083606362342834,
+      "learning_rate": 2.6142131979695434e-05,
+      "loss": 0.0921,
+      "step": 1375
+    },
+    {
+      "epoch": 4.367088607594937,
+      "grad_norm": 0.5320701599121094,
+      "learning_rate": 2.5507614213197974e-05,
+      "loss": 0.0917,
+      "step": 1380
+    },
+    {
+      "epoch": 4.382911392405063,
+      "grad_norm": 0.416904479265213,
+      "learning_rate": 2.4873096446700507e-05,
+      "loss": 0.0928,
+      "step": 1385
+    },
+    {
+      "epoch": 4.39873417721519,
+      "grad_norm": 0.49495264887809753,
+      "learning_rate": 2.4238578680203047e-05,
+      "loss": 0.0911,
+      "step": 1390
+    },
+    {
+      "epoch": 4.4145569620253164,
+      "grad_norm": 0.5663470029830933,
+      "learning_rate": 2.3604060913705588e-05,
+      "loss": 0.095,
+      "step": 1395
+    },
+    {
+      "epoch": 4.430379746835443,
+      "grad_norm": 0.49303102493286133,
+      "learning_rate": 2.296954314720812e-05,
+      "loss": 0.0937,
+      "step": 1400
+    },
+    {
+      "epoch": 4.44620253164557,
+      "grad_norm": 0.3795440196990967,
+      "learning_rate": 2.233502538071066e-05,
+      "loss": 0.0899,
+      "step": 1405
+    },
+    {
+      "epoch": 4.462025316455696,
+      "grad_norm": 0.4684379994869232,
+      "learning_rate": 2.17005076142132e-05,
+      "loss": 0.0924,
+      "step": 1410
+    },
+    {
+      "epoch": 4.477848101265823,
+      "grad_norm": 0.3904794156551361,
+      "learning_rate": 2.1065989847715735e-05,
+      "loss": 0.0922,
+      "step": 1415
+    },
+    {
+      "epoch": 4.493670886075949,
+      "grad_norm": 0.6037257313728333,
+      "learning_rate": 2.0431472081218275e-05,
+      "loss": 0.096,
+      "step": 1420
+    },
+    {
+      "epoch": 4.509493670886076,
+      "grad_norm": 0.30922073125839233,
+      "learning_rate": 1.9796954314720815e-05,
+      "loss": 0.091,
+      "step": 1425
+    },
+    {
+      "epoch": 4.525316455696203,
+      "grad_norm": 0.5561403036117554,
+      "learning_rate": 1.916243654822335e-05,
+      "loss": 0.0905,
+      "step": 1430
+    },
+    {
+      "epoch": 4.541139240506329,
+      "grad_norm": 0.389127641916275,
+      "learning_rate": 1.852791878172589e-05,
+      "loss": 0.0906,
+      "step": 1435
+    },
+    {
+      "epoch": 4.556962025316456,
+      "grad_norm": 0.38574787974357605,
+      "learning_rate": 1.789340101522843e-05,
+      "loss": 0.0891,
+      "step": 1440
+    },
+    {
+      "epoch": 4.572784810126582,
+      "grad_norm": 0.42243725061416626,
+      "learning_rate": 1.7258883248730966e-05,
+      "loss": 0.0961,
+      "step": 1445
+    },
+    {
+      "epoch": 4.588607594936709,
+      "grad_norm": 0.522402286529541,
+      "learning_rate": 1.6624365482233503e-05,
+      "loss": 0.0889,
+      "step": 1450
+    },
+    {
+      "epoch": 4.6044303797468356,
+      "grad_norm": 0.3562035858631134,
+      "learning_rate": 1.5989847715736043e-05,
+      "loss": 0.0879,
+      "step": 1455
+    },
+    {
+      "epoch": 4.620253164556962,
+      "grad_norm": 0.7289389967918396,
+      "learning_rate": 1.535532994923858e-05,
+      "loss": 0.0955,
+      "step": 1460
+    },
+    {
+      "epoch": 4.636075949367089,
+      "grad_norm": 0.4322461783885956,
+      "learning_rate": 1.4720812182741117e-05,
+      "loss": 0.0933,
+      "step": 1465
+    },
+    {
+      "epoch": 4.651898734177215,
+      "grad_norm": 0.5375872850418091,
+      "learning_rate": 1.4086294416243657e-05,
+      "loss": 0.0911,
+      "step": 1470
+    },
+    {
+      "epoch": 4.667721518987342,
+      "grad_norm": 0.4194104075431824,
+      "learning_rate": 1.3451776649746192e-05,
+      "loss": 0.0878,
+      "step": 1475
+    },
+    {
+      "epoch": 4.6835443037974684,
+      "grad_norm": 0.4603971242904663,
+      "learning_rate": 1.281725888324873e-05,
+      "loss": 0.0916,
+      "step": 1480
+    },
+    {
+      "epoch": 4.699367088607595,
+      "grad_norm": 0.34578830003738403,
+      "learning_rate": 1.218274111675127e-05,
+      "loss": 0.0905,
+      "step": 1485
+    },
+    {
+      "epoch": 4.715189873417722,
+      "grad_norm": 0.7133288383483887,
+      "learning_rate": 1.1548223350253808e-05,
+      "loss": 0.0889,
+      "step": 1490
+    },
+    {
+      "epoch": 4.731012658227848,
+      "grad_norm": 0.6632990837097168,
+      "learning_rate": 1.0913705583756344e-05,
+      "loss": 0.0912,
+      "step": 1495
+    },
+    {
+      "epoch": 4.746835443037975,
+      "grad_norm": 0.339555561542511,
+      "learning_rate": 1.0279187817258885e-05,
+      "loss": 0.091,
+      "step": 1500
+    },
+    {
+      "epoch": 4.762658227848101,
+      "grad_norm": 0.48535045981407166,
+      "learning_rate": 9.644670050761421e-06,
+      "loss": 0.0957,
+      "step": 1505
+    },
+    {
+      "epoch": 4.7784810126582276,
+      "grad_norm": 0.46517860889434814,
+      "learning_rate": 9.01015228426396e-06,
+      "loss": 0.0919,
+      "step": 1510
+    },
+    {
+      "epoch": 4.794303797468355,
+      "grad_norm": 0.6948912739753723,
+      "learning_rate": 8.375634517766498e-06,
+      "loss": 0.0899,
+      "step": 1515
+    },
+    {
+      "epoch": 4.810126582278481,
+      "grad_norm": 0.4421853721141815,
+      "learning_rate": 7.741116751269035e-06,
+      "loss": 0.092,
+      "step": 1520
+    },
+    {
+      "epoch": 4.825949367088608,
+      "grad_norm": 0.4198189973831177,
+      "learning_rate": 7.106598984771575e-06,
+      "loss": 0.0905,
+      "step": 1525
+    },
+    {
+      "epoch": 4.841772151898734,
+      "grad_norm": 0.47301727533340454,
+      "learning_rate": 6.472081218274112e-06,
+      "loss": 0.0907,
+      "step": 1530
+    },
+    {
+      "epoch": 4.8575949367088604,
+      "grad_norm": 0.36296045780181885,
+      "learning_rate": 5.83756345177665e-06,
+      "loss": 0.0894,
+      "step": 1535
+    },
+    {
+      "epoch": 4.8734177215189876,
+      "grad_norm": 0.4861876666545868,
+      "learning_rate": 5.203045685279188e-06,
+      "loss": 0.091,
+      "step": 1540
+    },
+    {
+      "epoch": 4.889240506329114,
+      "grad_norm": 0.44622302055358887,
+      "learning_rate": 4.568527918781726e-06,
+      "loss": 0.0937,
+      "step": 1545
+    },
+    {
+      "epoch": 4.905063291139241,
+      "grad_norm": 0.45360803604125977,
+      "learning_rate": 3.934010152284264e-06,
+      "loss": 0.0904,
+      "step": 1550
+    },
+    {
+      "epoch": 4.920886075949367,
+      "grad_norm": 0.5564279556274414,
+      "learning_rate": 3.2994923857868023e-06,
+      "loss": 0.0921,
+      "step": 1555
+    },
+    {
+      "epoch": 4.936708860759493,
+      "grad_norm": 0.5952951908111572,
+      "learning_rate": 2.6649746192893404e-06,
+      "loss": 0.0929,
+      "step": 1560
+    },
+    {
+      "epoch": 4.9525316455696204,
+      "grad_norm": 0.6146189570426941,
+      "learning_rate": 2.030456852791878e-06,
+      "loss": 0.0906,
+      "step": 1565
+    },
+    {
+      "epoch": 4.968354430379747,
+      "grad_norm": 0.5644097328186035,
+      "learning_rate": 1.3959390862944163e-06,
+      "loss": 0.0942,
+      "step": 1570
+    },
+    {
+      "epoch": 4.984177215189874,
+      "grad_norm": 0.43627479672431946,
+      "learning_rate": 7.614213197969544e-07,
+      "loss": 0.0935,
+      "step": 1575
+    },
+    {
+      "epoch": 5.0,
+      "grad_norm": 0.864841639995575,
+      "learning_rate": 1.2690355329949238e-07,
+      "loss": 0.0912,
+      "step": 1580
+    }
+  ],
+  "logging_steps": 5,
+  "max_steps": 1580,
+  "num_input_tokens_seen": 0,
+  "num_train_epochs": 5,
+  "save_steps": 500,
+  "stateful_callbacks": {
+    "TrainerControl": {
+      "args": {
+        "should_epoch_stop": false,
+        "should_evaluate": false,
+        "should_log": false,
+        "should_save": true,
+        "should_training_stop": true
+      },
+      "attributes": {}
+    }
+  },
+  "total_flos": 1.3721567654375424e+17,
+  "train_batch_size": 16,
+  "trial_name": null,
+  "trial_params": null
+}

experiments/de-en-pt-zh/checkpoint-1580/training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:dc1f965478a30d642e1d3280129a9cb365127c4096496511633d6b72e8ccf15e
+size 6289

experiments/de-en-pt-zh/checkpoint-500/README.md

@@ -0,0 +1,210 @@
+---
+base_model: unsloth/meta-llama-3.1-8b-instruct-bnb-4bit
+library_name: peft
+pipeline_tag: text-generation
+tags:
+- base_model:adapter:unsloth/meta-llama-3.1-8b-instruct-bnb-4bit
+- lora
+- sft
+- transformers
+- trl
+- unsloth
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.18.1

experiments/de-en-pt-zh/checkpoint-500/adapter_config.json

@@ -0,0 +1,50 @@
+{
+  "alora_invocation_tokens": null,
+  "alpha_pattern": {},
+  "arrow_config": null,
+  "auto_mapping": {
+    "base_model_class": "LlamaForCausalLM",
+    "parent_library": "transformers.models.llama.modeling_llama",
+    "unsloth_fixed": true
+  },
+  "base_model_name_or_path": "unsloth/meta-llama-3.1-8b-instruct-bnb-4bit",
+  "bias": "none",
+  "corda_config": null,
+  "ensure_weight_tying": false,
+  "eva_config": null,
+  "exclude_modules": null,
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 16,
+  "lora_bias": false,
+  "lora_dropout": 0.1,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "peft_version": "0.18.1",
+  "qalora_group_size": 16,
+  "r": 8,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "gate_proj",
+    "v_proj",
+    "o_proj",
+    "k_proj",
+    "up_proj",
+    "down_proj",
+    "q_proj"
+  ],
+  "target_parameters": null,
+  "task_type": "CAUSAL_LM",
+  "trainable_token_indices": null,
+  "use_dora": false,
+  "use_qalora": false,
+  "use_rslora": true
+}

experiments/de-en-pt-zh/checkpoint-500/adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:999c071c224c7037a29d77f5816ccfddb44b8242c7e930711f0a1b0a8aac5552
+size 83945296

experiments/de-en-pt-zh/checkpoint-500/chat_template.jinja

@@ -0,0 +1,109 @@
+{{- bos_token }}
+{%- if custom_tools is defined %}
+    {%- set tools = custom_tools %}
+{%- endif %}
+{%- if not tools_in_user_message is defined %}
+    {%- set tools_in_user_message = true %}
+{%- endif %}
+{%- if not date_string is defined %}
+    {%- set date_string = "26 Jul 2024" %}
+{%- endif %}
+{%- if not tools is defined %}
+    {%- set tools = none %}
+{%- endif %}
+
+{#- This block extracts the system message, so we can slot it into the right place. #}
+{%- if messages[0]['role'] == 'system' %}
+    {%- set system_message = messages[0]['content']|trim %}
+    {%- set messages = messages[1:] %}
+{%- else %}
+    {%- set system_message = "" %}
+{%- endif %}
+
+{#- System message + builtin tools #}
+{{- "<|start_header_id|>system<|end_header_id|>\n\n" }}
+{%- if builtin_tools is defined or tools is not none %}
+    {{- "Environment: ipython\n" }}
+{%- endif %}
+{%- if builtin_tools is defined %}
+    {{- "Tools: " + builtin_tools | reject('equalto', 'code_interpreter') | join(", ") + "\n\n"}}
+{%- endif %}
+{{- "Cutting Knowledge Date: December 2023\n" }}
+{{- "Today Date: " + date_string + "\n\n" }}
+{%- if tools is not none and not tools_in_user_message %}
+    {{- "You have access to the following functions. To call a function, please respond with JSON for a function call." }}
+    {{- 'Respond in the format {"name": function name, "parameters": dictionary of argument name and its value}.' }}
+    {{- "Do not use variables.\n\n" }}
+    {%- for t in tools %}
+        {{- t | tojson(indent=4) }}
+        {{- "\n\n" }}
+    {%- endfor %}
+{%- endif %}
+{{- system_message }}
+{{- "<|eot_id|>" }}
+
+{#- Custom tools are passed in a user message with some extra guidance #}
+{%- if tools_in_user_message and not tools is none %}
+    {#- Extract the first user message so we can plug it in here #}
+    {%- if messages | length != 0 %}
+        {%- set first_user_message = messages[0]['content']|trim %}
+        {%- set messages = messages[1:] %}
+    {%- else %}
+        {{- raise_exception("Cannot put tools in the first user message when there's no first user message!") }}
+{%- endif %}
+    {{- '<|start_header_id|>user<|end_header_id|>\n\n' -}}
+    {{- "Given the following functions, please respond with a JSON for a function call " }}
+    {{- "with its proper arguments that best answers the given prompt.\n\n" }}
+    {{- 'Respond in the format {"name": function name, "parameters": dictionary of argument name and its value}.' }}
+    {{- "Do not use variables.\n\n" }}
+    {%- for t in tools %}
+        {{- t | tojson(indent=4) }}
+        {{- "\n\n" }}
+    {%- endfor %}
+    {{- first_user_message + "<|eot_id|>"}}
+{%- endif %}
+
+{%- for message in messages %}
+    {%- if not (message.role == 'ipython' or message.role == 'tool' or 'tool_calls' in message) %}
+        {{- '<|start_header_id|>' + message['role'] + '<|end_header_id|>\n\n'+ message['content'] | trim + '<|eot_id|>' }}
+    {%- elif 'tool_calls' in message %}
+        {%- if not message.tool_calls|length == 1 %}
+            {{- raise_exception("This model only supports single tool-calls at once!") }}
+        {%- endif %}
+        {%- set tool_call = message.tool_calls[0].function %}
+        {%- if builtin_tools is defined and tool_call.name in builtin_tools %}
+            {{- '<|start_header_id|>assistant<|end_header_id|>\n\n' -}}
+            {{- "<|python_tag|>" + tool_call.name + ".call(" }}
+            {%- for arg_name, arg_val in tool_call.arguments | items %}
+                {{- arg_name + '="' + arg_val + '"' }}
+                {%- if not loop.last %}
+                    {{- ", " }}
+                {%- endif %}
+                {%- endfor %}
+            {{- ")" }}
+        {%- else  %}
+            {{- '<|start_header_id|>assistant<|end_header_id|>\n\n' -}}
+            {{- '{"name": "' + tool_call.name + '", ' }}
+            {{- '"parameters": ' }}
+            {{- tool_call.arguments | tojson }}
+            {{- "}" }}
+        {%- endif %}
+        {%- if builtin_tools is defined %}
+            {#- This means we're in ipython mode #}
+            {{- "<|eom_id|>" }}
+        {%- else %}
+            {{- "<|eot_id|>" }}
+        {%- endif %}
+    {%- elif message.role == "tool" or message.role == "ipython" %}
+        {{- "<|start_header_id|>ipython<|end_header_id|>\n\n" }}
+        {%- if message.content is mapping or message.content is iterable %}
+            {{- message.content | tojson }}
+        {%- else %}
+            {{- message.content }}
+        {%- endif %}
+        {{- "<|eot_id|>" }}
+    {%- endif %}
+{%- endfor %}
+{%- if add_generation_prompt %}
+    {{- '<|start_header_id|>assistant<|end_header_id|>\n\n' }}
+{%- endif %}

experiments/de-en-pt-zh/checkpoint-500/optimizer.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:48cc06e82e75cf7d4c8d7b0263a4aaf989b60be0a78d9a6c8d7afbf91447dd28
+size 43123429