Upload folder using huggingface_hub

.gitattributes

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+tokenizer.json filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,85 @@
+---
+license: llama3.2
+language:
+- en
+base_model: unsloth/Llama-3.2-1B-Instruct
+datasets:
+- m-a-p/SuperGPQA
+pipeline_tag: text-generation
+library_name: transformers
+tags:
+- sft
+- trl
+- unsloth
+- llama
+- llama3
+- llama3.1
+---
+![logo](https://i.imgur.com/iuUzm7L.jpeg)
+A fine-tune of [unsloth/Llama-3.2-1B-Instruct](https://huggingface.co/unsloth/Llama-3.2-1B-Instruct) on the [m-a-p/SuperGPQA](https://huggingface.co/datasets/m-a-p/SuperGPQA) dataset.
+
+## Usage example
+**System prompt**
+```
+You are a classifier. Categorize the following problem into discipline, field, and subfield in JSON format.
+```
+**User prompt**
+```
+Cotton and linen both readily catch fire. A batch of towels is composed of both cotton and linen, and is known to have caught fire. If it is known that the towels were ignited by a lit cigarette, which of the following arguments utilizes the most appropriate form of reasoning?
+```
+**Assistant response**
+```
+{"discipline": "Philosophy", "field": "Philosophy", "subfield": "Logic"}
+```
+# Possible output options
+Discipline
+```
+['Management', 'Philosophy', 'History', 'Engineering', 'Medicine', 'Literature and Arts', 'Economics', 'Science', 'Sociology', 'Agronomy', 'Education', 'Law', 'Military Science']
+```
+Field
+```
+['Geological Resources and Geological Engineering', 'Control Science and Engineering', 'Animal Husbandry', 'Chemistry', 'Agricultural Engineering', 'Veterinary Medicine', 'Systems Science', 'Petroleum and Natural Gas Engineering', 'Geography', 'Weapon Science and Technology', 'Surveying and Mapping Science and Technology', 'Atmospheric Science', 'Architecture', 'Transportation Engineering', 'Nuclear Science and Technology', 'Forestry Engineering', 'Business Administration', 'Optical Engineering', 'Traditional Chinese Medicine', 'Musicology', 'Philosophy', 'Electronic Science and Technology', 'Physical Education', 'Language and Literature', 'Library, Information and Archival Management', 'Electrical Engineering', 'Art Studies', 'Civil Engineering', 'Stomatology', 'Crop Science', 'Instrument Science and Technology', 'Sociology', 'Education', 'Chemical Engineering and Technology', 'Aquaculture', 'Clinical Medicine', 'Computer Science and Technology', 'Physics', 'Aeronautical and Astronautical Science and Technology', 'Military Science', 'Mechanics', 'Environmental Science and Engineering', 'Food Science and Engineering', 'Management Science and Engineering', 'Psychology', 'History', 'Mining Engineering', 'Power Engineering and Engineering Thermophysics', 'Hydraulic Engineering', 'Mechanical Engineering', 'Physical Oceanography', 'Applied Economics', 'Metallurgical Engineering', 'Mathematics', 'Political Science', 'Basic Medicine', 'Geophysics', 'Public Administration', 'Geology', 'Forestry', 'Oceanography', 'Theoretical Economics', 'Journalism and Communication', 'Textile Science and Engineering', 'Information and Communication Engineering', 'Materials Science and Engineering', 'Biology', 'Naval Architecture and Ocean Engineering', 'Public Health and Preventive Medicine', 'Law', 'Pharmacy', 'Astronomy']
+```
+Subfield
+```
+['Mineral Processing Engineering', 'Labor Economics', 'Instrumentation and Performance', 'Otorhinolaryngology', 'Analytical Chemistry', 'Guidance, Navigation and Control', 'Linguistics and Applied Linguistics', 'Military Command and Information Systems', 'Mining and Safety Engineering', 'Forest Cultivation and Genetic Breeding', 'Obstetrics and Gynecology', 'Physical Chemistry', 'Signal and Information Processing', 'Marine Engineering', 'Nutrition and Food Hygiene', 'Theoretical Mechanics', 'Education Economics, Management and Social Security', 'Transportation Planning and Management', 'Quantitative Economics', 'Atomic and Molecular Physics', 'Fundamental Mathematics', 'Imaging and Nuclear Medicine', 'Subatomic and Atomic Physics', 'Rigid Body Mechanics', 'Electrodynamics', 'Crop Science', 'Radiochemistry', 'Group Theory', 'National and Defense Economics', 'Civil and Commercial Law', 'Logic', 'Internal Combustion Engineering', 'Aquaculture', 'Sports Science and Medicine', 'Dance Studies', 'Aeronautical and Astronautical Science and Technology', 'Ordinary Differential Equations', 'Materials Physics and Chemistry', 'Educational Technology and Principles', 'Theoretical Optics', 'Physical Chemistry of Metallurgical Process', 'Solid Mechanics', 'Urban Planning and Design', 'Botany', 'Databases', 'Wood Science and Technology', 'Maternal, Child and Adolescent Health', 'Atmospheric Physics and Atmospheric Environment', 'Literary History', 'Polynomials and Series Expansions', 'Constitutional and Administrative Law', 'Political Science', 'Discrete Mathematics', 'Vehicle Operation Engineering', 'Fluid Machinery and Engineering', 'Harmony', 'Philosophy of Science and Technology', 'Functions of Complex Variables', 'Electrochemistry', 'Internal Medicine', 'Statistical Mechanics', 'Pathogen Biology', 'Hydrogeology', 'Agricultural Mechanization Engineering', 'Fundamentals of Dynamics and Control', 'Clinical Stomatology', 'Antenna and Radio Communication', 'Military Management', 'Electrical Theory and New Technologies', 'Power Systems and Automation', 'Food Biochemistry', 'Formal Languages', 'Water conservancy and Hydropower Engineering', 'Legal Theory and Legal History', 'Optical Fiber Communication', 'Astrophysics', 'Contract Law', 'Preschool Education', 'Power Machinery and Engineering', 'Advanced Programming Languages', 'Principles of Metallurgy', 'Astronomical Observation and Technology', 'Physiology', 'Advanced Algebra', 'Economic Statistics', 'Veterinary Medicine', 'Anesthesiology', 'Psychiatry and Mental Health', 'Military Logistics and Equipment', 'Electromagnetic Field and Microwave Technology', 'Music History, Education, and Technology', 'Meteorology', 'Engineering Fluid Mechanics', 'Road and Railway Engineering', 'Composition', 'Forensic Medicine', 'Geotechnical Engineering', 'Pitch and Scales', 'Environmental and Resource Protection', 'Historical Geography', 'Communication Principles', 'Dynamic Meteorology', 'Mechatronic Engineering', 'Environmental Science', 'Theoretical Fluid Mechanics', 'Circuits and Systems', 'Archaeology and Museology', 'Geriatric Medicine', 'Land Resource Management and Administrative Management', 'Weapon Systems Science and Engineering', 'Marine Biology', 'Particle and Nuclear Physics', 'Biochemistry and Molecular Biology', 'Procedural Law', 'High Voltage and Insulation Technology', 'Philology and Bibliography', 'Design Arts', 'Optoelectronic Technology', 'Poromechanics and Reservoir Physics', 'Engineering Thermophysics', 'Broadcasting and Television Art', 'Library and Archival Science', 'Ethics', 'Physical Oceanography', 'Solar System Science', 'Thermal Energy Engineering', 'Fuzzy Mathematics', 'Agricultural Environment and Soil-Water Engineering', 'Cartography and Geographic Information Engineering', 'Genetics', 'Chemical Transport Engineering', 'Film Studies', 'Information Management Science', 'Architectural Design and Theory', 'Pharmacology', 'Computer Software and Theory', 'Inorganic Chemistry', 'Microbiology', 'Philosophical Aesthetics', 'Non-ferrous Metallurgy', 'Stellar and Interstellar Evolution', 'Computer Networks', 'Marine Chemistry', 'Thermodynamics and Statistical Physics', 'Operating Systems', 'Elements of Chemical Reaction Engineering', 'Thermodynamics', 'Literary Theory', 'Musical Forms and Analysis', 'Traditional Chinese Health Preservation', 'Geological Resources and Geological Engineering', 'Surgery', 'Operations Research and Cybernetics', 'Journalism and News Practice', 'Drama and Opera Studies', 'Human Geography', 'Systems Science', 'Industrial Economics', 'Organic Chemistry', 'Social and Folklore Studies', 'Bridge and Tunnel Engineering', 'Military Chemistry and Pyrotechnics', 'Principles of Seismic Exploration', 'Landscape Plants and Ornamental Horticulture', 'Communication and Information Systems', 'Traditional Chinese Pharmacy', 'Finance', 'Computational Mathematics', 'History and Theory of Journalism and Media Management', 'Emergency Medicine', 'Digital Surveying and Remote Sensing Applications', 'Quantum Mechanics', 'Nuclear Energy and Reactor Technology', 'Mathematical Analysis', 'Biophysics', 'Microelectronics and Solid-State Electronics', 'International Law', 'Mass Transport and Separation Process in Chemical Engineering', 'Medicinal Chemistry', 'Physical Education and Training', 'Forest Engineering', 'Instrument Science and Technology', 'Laser Technology', 'Information Management and Communication', 'Russian Language and Literature', 'Theory of Curriculum and Instruction', 'Power Electronics and Electrical Drives', 'Cryptography', 'Communication and Broadcasting', 'Functions of Real Variables', 'International Trade', 'Materials Processing Engineering', 'Economic History', 'Traffic Information Engineering and Control', 'Textile Chemistry and Dyeing Engineering', 'Urban Infrastructure Engineering', 'Semiconductor Physics', 'Ship Mechanics and Design Principles', 'Neurology', 'Immunology', 'Ophthalmology', 'Ecology', 'Heat Transfer', 'Graph Theory', 'Epidemiology and Health Statistics', 'Military Thought and History', 'Radiation Protection and Nuclear Technology Applications', 'Computer Architecture', 'Control Theory and Control Engineering', 'Polymer Physics', 'Underwater Acoustics', 'Nursing and Rehabilitation Medicine', 'Radiation Medicine', 'Geochemistry', 'Traditional Chinese Medicine Theory', 'Oncology', 'Physical Geography', 'Tourism Management and Technological Economics Management', 'Number Theory', 'Social Medicine and Health Management', 'Pediatrics', 'French Language and Literature', 'Demography and Anthropology', 'Oil and Gas Field Development and Storage & Transportation Engineering', 'Zoology', 'Sports Humanities and Sociology', 'Architectural History', 'Environmental Engineering', 'Paleontology and Stratigraphy', 'Space physics', 'Public Finance', 'Structural Geology', 'Combinatorial Mathematics', 'Applied Optics', 'Food Processing and Storage Engineering', 'Iron and Steel Metallurgy', 'Fluid Physics', 'Hydraulics and Hydrology', 'World History', 'Classical Chinese Literature', 'Solid State Physics', 'Textile Materials Science', 'Dermatology and Venereology', 'Geometry and Topology', 'Data Structures', 'Military Law', 'Health Toxicology and Environmental Health', 'Religious Studies', 'Management Science and Engineering', 'Relativity', 'Psychology', 'Criminal Law', 'Refrigeration and Cryogenic Engineering', 'Political Economy', 'Stochastic Processes', 'Clinical Laboratory Diagnostics', 'Cell Biology', 'Modern and Contemporary Chinese Literature', 'Western Economics', 'Basic Stomatology', 'Solid Earth Geophysics', 'Geodesy and Surveying Engineering', 'Microbiology and Biochemical Pharmacy', 'Pattern Recognition', 'Animal Rearing and Breeding', 'Polymer Chemistry and Physics', 'Acoustics', 'Business and Accounting Management', 'Fluid Flow and Heat Transfer in Chemical Engineering', 'Cosmology', 'Mineralogy, Petrology, and Economic Geology', 'Special Education', 'Principles of Computer Organization', 'Fine Arts', 'Structural Engineering', 'Pharmaceutical Analysis', 'Probability and Statistics', 'Special Number Theory', 'Animal Nutrition and Feed Science', 'Pharmaceutics', 'Manufacturing Automation', 'Human Anatomy and Histology-Embryology', 'Law and Social Governance', 'Pathology and Pathophysiology', 'Numerical Analysis']
+```
+## Model Details
+- Base Model: `unsloth/Llama-3.2-1B-Instruct`
+- Parameter Count: 1235814400
+- Precision: torch.bfloat16
+
+## Training Settings
+### Hardware
+- GPU: NVIDIA A100-SXM4-40GB
+
+### PEFT
+- Rank: 32
+- LoRA alpha: 64
+- Modules: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
+
+### SFT
+- Epoch: 2
+- Batch size: 16
+- Gradient Accumulation steps: 1
+- Warmup ratio: 0.1
+- Learning rate: 0.0002
+- Optimizer: adamw_torch_fused
+- Learning rate scheduler: cosine
+
+## Training stats
+- Global step: 3152
+- Training runtime (seconds): 1163.1262
+- Average training loss: 0.0777996052304746
+- Final validation loss: 0.051274195313453674
+
+## Framework versions
+- Unsloth: 2026.3.8
+- TRL: 0.22.2
+- Transformers: 4.56.2
+- Pytorch: 2.10.0+cu128
+- Datasets: 4.3.0
+- Tokenizers: 0.22.2
+
+## License
+This model is released under the Llama3 license. See the [Terms of Use](https://www.llama.com/llama3/license/) for details.

chat_template.jinja

@@ -0,0 +1,93 @@
+{{- bos_token }}
+{%- if custom_tools is defined %}
+    {%- set tools = custom_tools %}
+{%- endif %}
+{%- if not tools_in_user_message is defined %}
+    {%- set tools_in_user_message = true %}
+{%- endif %}
+{%- if not date_string is defined %}
+    {%- if strftime_now is defined %}
+        {%- set date_string = strftime_now("%d %b %Y") %}
+    {%- else %}
+        {%- set date_string = "26 Jul 2024" %}
+    {%- endif %}
+{%- endif %}
+{%- if not tools is defined %}
+    {%- set tools = none %}
+{%- endif %}
+
+{#- This block extracts the system message, so we can slot it into the right place. #}
+{%- if messages[0]['role'] == 'system' %}
+    {%- set system_message = messages[0]['content']|trim %}
+    {%- set messages = messages[1:] %}
+{%- else %}
+    {%- set system_message = "" %}
+{%- endif %}
+
+{#- System message #}
+{{- "<|start_header_id|>system<|end_header_id|>\n\n" }}
+{%- if tools is not none %}
+    {{- "Environment: ipython\n" }}
+{%- endif %}
+{{- "Cutting Knowledge Date: December 2023\n" }}
+{{- "Today Date: " + date_string + "\n\n" }}
+{%- if tools is not none and not tools_in_user_message %}
+    {{- "You have access to the following functions. To call a function, please respond with JSON for a function call." }}
+    {{- 'Respond in the format {"name": function name, "parameters": dictionary of argument name and its value}.' }}
+    {{- "Do not use variables.\n\n" }}
+    {%- for t in tools %}
+        {{- t | tojson(indent=4) }}
+        {{- "\n\n" }}
+    {%- endfor %}
+{%- endif %}
+{{- system_message }}
+{{- "<|eot_id|>" }}
+
+{#- Custom tools are passed in a user message with some extra guidance #}
+{%- if tools_in_user_message and not tools is none %}
+    {#- Extract the first user message so we can plug it in here #}
+    {%- if messages | length != 0 %}
+        {%- set first_user_message = messages[0]['content']|trim %}
+        {%- set messages = messages[1:] %}
+    {%- else %}
+        {{- raise_exception("Cannot put tools in the first user message when there's no first user message!") }}
+{%- endif %}
+    {{- '<|start_header_id|>user<|end_header_id|>\n\n' -}}
+    {{- "Given the following functions, please respond with a JSON for a function call " }}
+    {{- "with its proper arguments that best answers the given prompt.\n\n" }}
+    {{- 'Respond in the format {"name": function name, "parameters": dictionary of argument name and its value}.' }}
+    {{- "Do not use variables.\n\n" }}
+    {%- for t in tools %}
+        {{- t | tojson(indent=4) }}
+        {{- "\n\n" }}
+    {%- endfor %}
+    {{- first_user_message + "<|eot_id|>"}}
+{%- endif %}
+
+{%- for message in messages %}
+    {%- if not (message.role == 'ipython' or message.role == 'tool' or 'tool_calls' in message) %}
+        {{- '<|start_header_id|>' + message['role'] + '<|end_header_id|>\n\n'+ message['content'] | trim + '<|eot_id|>' }}
+    {%- elif 'tool_calls' in message %}
+        {%- if not message.tool_calls|length == 1 %}
+            {{- raise_exception("This model only supports single tool-calls at once!") }}
+        {%- endif %}
+        {%- set tool_call = message.tool_calls[0].function %}
+        {{- '<|start_header_id|>assistant<|end_header_id|>\n\n' -}}
+        {{- '{"name": "' + tool_call.name + '", ' }}
+        {{- '"parameters": ' }}
+        {{- tool_call.arguments | tojson }}
+        {{- "}" }}
+        {{- "<|eot_id|>" }}
+    {%- elif message.role == "tool" or message.role == "ipython" %}
+        {{- "<|start_header_id|>ipython<|end_header_id|>\n\n" }}
+        {%- if message.content is mapping or message.content is iterable %}
+            {{- message.content | tojson }}
+        {%- else %}
+            {{- message.content }}
+        {%- endif %}
+        {{- "<|eot_id|>" }}
+    {%- endif %}
+{%- endfor %}
+{%- if add_generation_prompt %}
+    {{- '<|start_header_id|>assistant<|end_header_id|>\n\n' }}
+{%- endif %}

config.json

@@ -0,0 +1,37 @@
+{
+  "architectures": [
+    "LlamaForCausalLM"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "bos_token_id": 128000,
+  "dtype": "bfloat16",
+  "eos_token_id": 128009,
+  "head_dim": 64,
+  "hidden_act": "silu",
+  "hidden_size": 2048,
+  "initializer_range": 0.02,
+  "intermediate_size": 8192,
+  "max_position_embeddings": 131072,
+  "mlp_bias": false,
+  "model_type": "llama",
+  "num_attention_heads": 32,
+  "num_hidden_layers": 16,
+  "num_key_value_heads": 8,
+  "pad_token_id": 128004,
+  "pretraining_tp": 1,
+  "rms_norm_eps": 1e-05,
+  "rope_scaling": {
+    "factor": 32.0,
+    "high_freq_factor": 4.0,
+    "low_freq_factor": 1.0,
+    "original_max_position_embeddings": 8192,
+    "rope_type": "llama3"
+  },
+  "rope_theta": 500000.0,
+  "tie_word_embeddings": true,
+  "transformers_version": "4.56.2",
+  "unsloth_fixed": true,
+  "use_cache": true,
+  "vocab_size": 128256
+}

generation_config.json

@@ -0,0 +1,14 @@
+{
+  "bos_token_id": 128000,
+  "do_sample": true,
+  "eos_token_id": [
+    128001,
+    128008,
+    128009
+  ],
+  "max_length": 131072,
+  "pad_token_id": 128004,
+  "temperature": 0.6,
+  "top_p": 0.9,
+  "transformers_version": "4.56.2"
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:57401189f8100bbc8a83e3731582721ac23c46c964245a881f675c3858b61324
+size 2471645608

special_tokens_map.json

@@ -0,0 +1,23 @@
+{
+  "bos_token": {
+    "content": "<|begin_of_text|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "<|eot_id|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "<|finetune_right_pad_id|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

tokenizer.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6b9e4e7fb171f92fd137b777cc2714bf87d11576700a1dcd7a399e7bbe39537b
+size 17209920

tokenizer_config.json

@@ -0,0 +1,2066 @@
+{
+  "add_bos_token": true,
+  "added_tokens_decoder": {
+    "128000": {
+      "content": "<|begin_of_text|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128001": {
+      "content": "<|end_of_text|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128002": {
+      "content": "<|reserved_special_token_0|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128003": {
+      "content": "<|reserved_special_token_1|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128004": {
+      "content": "<|finetune_right_pad_id|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128005": {
+      "content": "<|reserved_special_token_2|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128006": {
+      "content": "<|start_header_id|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128007": {
+      "content": "<|end_header_id|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128008": {
+      "content": "<|eom_id|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128009": {
+      "content": "<|eot_id|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128010": {
+      "content": "<|python_tag|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128011": {
+      "content": "<|reserved_special_token_3|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128012": {
+      "content": "<|reserved_special_token_4|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128013": {
+      "content": "<|reserved_special_token_5|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128014": {
+      "content": "<|reserved_special_token_6|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128015": {
+      "content": "<|reserved_special_token_7|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128016": {
+      "content": "<|reserved_special_token_8|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128017": {
+      "content": "<|reserved_special_token_9|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128018": {
+      "content": "<|reserved_special_token_10|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128019": {
+      "content": "<|reserved_special_token_11|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128020": {
+      "content": "<|reserved_special_token_12|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128021": {
+      "content": "<|reserved_special_token_13|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128022": {
+      "content": "<|reserved_special_token_14|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128023": {
+      "content": "<|reserved_special_token_15|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128024": {
+      "content": "<|reserved_special_token_16|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128025": {
+      "content": "<|reserved_special_token_17|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128026": {
+      "content": "<|reserved_special_token_18|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128027": {
+      "content": "<|reserved_special_token_19|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128028": {
+      "content": "<|reserved_special_token_20|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128029": {
+      "content": "<|reserved_special_token_21|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128030": {
+      "content": "<|reserved_special_token_22|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128031": {
+      "content": "<|reserved_special_token_23|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128032": {
+      "content": "<|reserved_special_token_24|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128033": {
+      "content": "<|reserved_special_token_25|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128034": {
+      "content": "<|reserved_special_token_26|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128035": {
+      "content": "<|reserved_special_token_27|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128036": {
+      "content": "<|reserved_special_token_28|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128037": {
+      "content": "<|reserved_special_token_29|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128038": {
+      "content": "<|reserved_special_token_30|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128039": {
+      "content": "<|reserved_special_token_31|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128040": {
+      "content": "<|reserved_special_token_32|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128041": {
+      "content": "<|reserved_special_token_33|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128042": {
+      "content": "<|reserved_special_token_34|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128043": {
+      "content": "<|reserved_special_token_35|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128044": {
+      "content": "<|reserved_special_token_36|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128045": {
+      "content": "<|reserved_special_token_37|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128046": {
+      "content": "<|reserved_special_token_38|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128047": {
+      "content": "<|reserved_special_token_39|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128048": {
+      "content": "<|reserved_special_token_40|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128049": {
+      "content": "<|reserved_special_token_41|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128050": {
+      "content": "<|reserved_special_token_42|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128051": {
+      "content": "<|reserved_special_token_43|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128052": {
+      "content": "<|reserved_special_token_44|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128053": {
+      "content": "<|reserved_special_token_45|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128054": {
+      "content": "<|reserved_special_token_46|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128055": {
+      "content": "<|reserved_special_token_47|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128056": {
+      "content": "<|reserved_special_token_48|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128057": {
+      "content": "<|reserved_special_token_49|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128058": {
+      "content": "<|reserved_special_token_50|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128059": {
+      "content": "<|reserved_special_token_51|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128060": {
+      "content": "<|reserved_special_token_52|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128061": {
+      "content": "<|reserved_special_token_53|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128062": {
+      "content": "<|reserved_special_token_54|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128063": {
+      "content": "<|reserved_special_token_55|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128064": {
+      "content": "<|reserved_special_token_56|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128065": {
+      "content": "<|reserved_special_token_57|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128066": {
+      "content": "<|reserved_special_token_58|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128067": {
+      "content": "<|reserved_special_token_59|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128068": {
+      "content": "<|reserved_special_token_60|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128069": {
+      "content": "<|reserved_special_token_61|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128070": {
+      "content": "<|reserved_special_token_62|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128071": {
+      "content": "<|reserved_special_token_63|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128072": {
+      "content": "<|reserved_special_token_64|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128073": {
+      "content": "<|reserved_special_token_65|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128074": {
+      "content": "<|reserved_special_token_66|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128075": {
+      "content": "<|reserved_special_token_67|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128076": {
+      "content": "<|reserved_special_token_68|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128077": {
+      "content": "<|reserved_special_token_69|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128078": {
+      "content": "<|reserved_special_token_70|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128079": {
+      "content": "<|reserved_special_token_71|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128080": {
+      "content": "<|reserved_special_token_72|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128081": {
+      "content": "<|reserved_special_token_73|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128082": {
+      "content": "<|reserved_special_token_74|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128083": {
+      "content": "<|reserved_special_token_75|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128084": {
+      "content": "<|reserved_special_token_76|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128085": {
+      "content": "<|reserved_special_token_77|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128086": {
+      "content": "<|reserved_special_token_78|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128087": {
+      "content": "<|reserved_special_token_79|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128088": {
+      "content": "<|reserved_special_token_80|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128089": {
+      "content": "<|reserved_special_token_81|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128090": {
+      "content": "<|reserved_special_token_82|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128091": {
+      "content": "<|reserved_special_token_83|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128092": {
+      "content": "<|reserved_special_token_84|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128093": {
+      "content": "<|reserved_special_token_85|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128094": {
+      "content": "<|reserved_special_token_86|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128095": {
+      "content": "<|reserved_special_token_87|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128096": {
+      "content": "<|reserved_special_token_88|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128097": {
+      "content": "<|reserved_special_token_89|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128098": {
+      "content": "<|reserved_special_token_90|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128099": {
+      "content": "<|reserved_special_token_91|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128100": {
+      "content": "<|reserved_special_token_92|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128101": {
+      "content": "<|reserved_special_token_93|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128102": {
+      "content": "<|reserved_special_token_94|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128103": {
+      "content": "<|reserved_special_token_95|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128104": {
+      "content": "<|reserved_special_token_96|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128105": {
+      "content": "<|reserved_special_token_97|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128106": {
+      "content": "<|reserved_special_token_98|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128107": {
+      "content": "<|reserved_special_token_99|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128108": {
+      "content": "<|reserved_special_token_100|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128109": {
+      "content": "<|reserved_special_token_101|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128110": {
+      "content": "<|reserved_special_token_102|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128111": {
+      "content": "<|reserved_special_token_103|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128112": {
+      "content": "<|reserved_special_token_104|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128113": {
+      "content": "<|reserved_special_token_105|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128114": {
+      "content": "<|reserved_special_token_106|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128115": {
+      "content": "<|reserved_special_token_107|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128116": {
+      "content": "<|reserved_special_token_108|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128117": {
+      "content": "<|reserved_special_token_109|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128118": {
+      "content": "<|reserved_special_token_110|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128119": {
+      "content": "<|reserved_special_token_111|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128120": {
+      "content": "<|reserved_special_token_112|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128121": {
+      "content": "<|reserved_special_token_113|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128122": {
+      "content": "<|reserved_special_token_114|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128123": {
+      "content": "<|reserved_special_token_115|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128124": {
+      "content": "<|reserved_special_token_116|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128125": {
+      "content": "<|reserved_special_token_117|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128126": {
+      "content": "<|reserved_special_token_118|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128127": {
+      "content": "<|reserved_special_token_119|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128128": {
+      "content": "<|reserved_special_token_120|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128129": {
+      "content": "<|reserved_special_token_121|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128130": {
+      "content": "<|reserved_special_token_122|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128131": {
+      "content": "<|reserved_special_token_123|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128132": {
+      "content": "<|reserved_special_token_124|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128133": {
+      "content": "<|reserved_special_token_125|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128134": {
+      "content": "<|reserved_special_token_126|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128135": {
+      "content": "<|reserved_special_token_127|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128136": {
+      "content": "<|reserved_special_token_128|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128137": {
+      "content": "<|reserved_special_token_129|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128138": {
+      "content": "<|reserved_special_token_130|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128139": {
+      "content": "<|reserved_special_token_131|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128140": {
+      "content": "<|reserved_special_token_132|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128141": {
+      "content": "<|reserved_special_token_133|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128142": {
+      "content": "<|reserved_special_token_134|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128143": {
+      "content": "<|reserved_special_token_135|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128144": {
+      "content": "<|reserved_special_token_136|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128145": {
+      "content": "<|reserved_special_token_137|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128146": {
+      "content": "<|reserved_special_token_138|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128147": {
+      "content": "<|reserved_special_token_139|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128148": {
+      "content": "<|reserved_special_token_140|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128149": {
+      "content": "<|reserved_special_token_141|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128150": {
+      "content": "<|reserved_special_token_142|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128151": {
+      "content": "<|reserved_special_token_143|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128152": {
+      "content": "<|reserved_special_token_144|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128153": {
+      "content": "<|reserved_special_token_145|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128154": {
+      "content": "<|reserved_special_token_146|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128155": {
+      "content": "<|reserved_special_token_147|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128156": {
+      "content": "<|reserved_special_token_148|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128157": {
+      "content": "<|reserved_special_token_149|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128158": {
+      "content": "<|reserved_special_token_150|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128159": {
+      "content": "<|reserved_special_token_151|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128160": {
+      "content": "<|reserved_special_token_152|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128161": {
+      "content": "<|reserved_special_token_153|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128162": {
+      "content": "<|reserved_special_token_154|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128163": {
+      "content": "<|reserved_special_token_155|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128164": {
+      "content": "<|reserved_special_token_156|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128165": {
+      "content": "<|reserved_special_token_157|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128166": {
+      "content": "<|reserved_special_token_158|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128167": {
+      "content": "<|reserved_special_token_159|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128168": {
+      "content": "<|reserved_special_token_160|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128169": {
+      "content": "<|reserved_special_token_161|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128170": {
+      "content": "<|reserved_special_token_162|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128171": {
+      "content": "<|reserved_special_token_163|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128172": {
+      "content": "<|reserved_special_token_164|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128173": {
+      "content": "<|reserved_special_token_165|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128174": {
+      "content": "<|reserved_special_token_166|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128175": {
+      "content": "<|reserved_special_token_167|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128176": {
+      "content": "<|reserved_special_token_168|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128177": {
+      "content": "<|reserved_special_token_169|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128178": {
+      "content": "<|reserved_special_token_170|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128179": {
+      "content": "<|reserved_special_token_171|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128180": {
+      "content": "<|reserved_special_token_172|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128181": {
+      "content": "<|reserved_special_token_173|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128182": {
+      "content": "<|reserved_special_token_174|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128183": {
+      "content": "<|reserved_special_token_175|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128184": {
+      "content": "<|reserved_special_token_176|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128185": {
+      "content": "<|reserved_special_token_177|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128186": {
+      "content": "<|reserved_special_token_178|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128187": {
+      "content": "<|reserved_special_token_179|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128188": {
+      "content": "<|reserved_special_token_180|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128189": {
+      "content": "<|reserved_special_token_181|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128190": {
+      "content": "<|reserved_special_token_182|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128191": {
+      "content": "<|reserved_special_token_183|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128192": {
+      "content": "<|reserved_special_token_184|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128193": {
+      "content": "<|reserved_special_token_185|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128194": {
+      "content": "<|reserved_special_token_186|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128195": {
+      "content": "<|reserved_special_token_187|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128196": {
+      "content": "<|reserved_special_token_188|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128197": {
+      "content": "<|reserved_special_token_189|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128198": {
+      "content": "<|reserved_special_token_190|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128199": {
+      "content": "<|reserved_special_token_191|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128200": {
+      "content": "<|reserved_special_token_192|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128201": {
+      "content": "<|reserved_special_token_193|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128202": {
+      "content": "<|reserved_special_token_194|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128203": {
+      "content": "<|reserved_special_token_195|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128204": {
+      "content": "<|reserved_special_token_196|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128205": {
+      "content": "<|reserved_special_token_197|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128206": {
+      "content": "<|reserved_special_token_198|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128207": {
+      "content": "<|reserved_special_token_199|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128208": {
+      "content": "<|reserved_special_token_200|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128209": {
+      "content": "<|reserved_special_token_201|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128210": {
+      "content": "<|reserved_special_token_202|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128211": {
+      "content": "<|reserved_special_token_203|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128212": {
+      "content": "<|reserved_special_token_204|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128213": {
+      "content": "<|reserved_special_token_205|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128214": {
+      "content": "<|reserved_special_token_206|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128215": {
+      "content": "<|reserved_special_token_207|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128216": {
+      "content": "<|reserved_special_token_208|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128217": {
+      "content": "<|reserved_special_token_209|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128218": {
+      "content": "<|reserved_special_token_210|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128219": {
+      "content": "<|reserved_special_token_211|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128220": {
+      "content": "<|reserved_special_token_212|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128221": {
+      "content": "<|reserved_special_token_213|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128222": {
+      "content": "<|reserved_special_token_214|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128223": {
+      "content": "<|reserved_special_token_215|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128224": {
+      "content": "<|reserved_special_token_216|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128225": {
+      "content": "<|reserved_special_token_217|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128226": {
+      "content": "<|reserved_special_token_218|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128227": {
+      "content": "<|reserved_special_token_219|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128228": {
+      "content": "<|reserved_special_token_220|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128229": {
+      "content": "<|reserved_special_token_221|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128230": {
+      "content": "<|reserved_special_token_222|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128231": {
+      "content": "<|reserved_special_token_223|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128232": {
+      "content": "<|reserved_special_token_224|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128233": {
+      "content": "<|reserved_special_token_225|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128234": {
+      "content": "<|reserved_special_token_226|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128235": {
+      "content": "<|reserved_special_token_227|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128236": {
+      "content": "<|reserved_special_token_228|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128237": {
+      "content": "<|reserved_special_token_229|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128238": {
+      "content": "<|reserved_special_token_230|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128239": {
+      "content": "<|reserved_special_token_231|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128240": {
+      "content": "<|reserved_special_token_232|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128241": {
+      "content": "<|reserved_special_token_233|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128242": {
+      "content": "<|reserved_special_token_234|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128243": {
+      "content": "<|reserved_special_token_235|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128244": {
+      "content": "<|reserved_special_token_236|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128245": {
+      "content": "<|reserved_special_token_237|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128246": {
+      "content": "<|reserved_special_token_238|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128247": {
+      "content": "<|reserved_special_token_239|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128248": {
+      "content": "<|reserved_special_token_240|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128249": {
+      "content": "<|reserved_special_token_241|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128250": {
+      "content": "<|reserved_special_token_242|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128251": {
+      "content": "<|reserved_special_token_243|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128252": {
+      "content": "<|reserved_special_token_244|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128253": {
+      "content": "<|reserved_special_token_245|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128254": {
+      "content": "<|reserved_special_token_246|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128255": {
+      "content": "<|reserved_special_token_247|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "bos_token": "<|begin_of_text|>",
+  "clean_up_tokenization_spaces": true,
+  "eos_token": "<|eot_id|>",
+  "extra_special_tokens": {},
+  "model_input_names": [
+    "input_ids",
+    "attention_mask"
+  ],
+  "model_max_length": 131072,
+  "pad_token": "<|finetune_right_pad_id|>",
+  "padding_side": "left",
+  "tokenizer_class": "PreTrainedTokenizerFast",
+  "unk_token": null
+}

train/log.json

@@ -0,0 +1,2296 @@
+[
+  {
+    "loss": 1.416,
+    "grad_norm": 7.261270046234131,
+    "learning_rate": 5.69620253164557e-06,
+    "epoch": 0.006345177664974619,
+    "step": 10
+  },
+  {
+    "loss": 1.0241,
+    "grad_norm": 4.087796688079834,
+    "learning_rate": 1.2025316455696203e-05,
+    "epoch": 0.012690355329949238,
+    "step": 20
+  },
+  {
+    "loss": 0.7054,
+    "grad_norm": 3.7453219890594482,
+    "learning_rate": 1.8354430379746836e-05,
+    "epoch": 0.01903553299492386,
+    "step": 30
+  },
+  {
+    "loss": 0.5727,
+    "grad_norm": 1.5524132251739502,
+    "learning_rate": 2.468354430379747e-05,
+    "epoch": 0.025380710659898477,
+    "step": 40
+  },
+  {
+    "loss": 0.4431,
+    "grad_norm": 1.175743579864502,
+    "learning_rate": 3.10126582278481e-05,
+    "epoch": 0.031725888324873094,
+    "step": 50
+  },
+  {
+    "loss": 0.3589,
+    "grad_norm": 1.3493213653564453,
+    "learning_rate": 3.7341772151898736e-05,
+    "epoch": 0.03807106598984772,
+    "step": 60
+  },
+  {
+    "loss": 0.3363,
+    "grad_norm": 1.4607553482055664,
+    "learning_rate": 4.367088607594937e-05,
+    "epoch": 0.044416243654822336,
+    "step": 70
+  },
+  {
+    "loss": 0.2936,
+    "grad_norm": 1.7902618646621704,
+    "learning_rate": 5e-05,
+    "epoch": 0.050761421319796954,
+    "step": 80
+  },
+  {
+    "loss": 0.27,
+    "grad_norm": 1.5136945247650146,
+    "learning_rate": 5.6329113924050636e-05,
+    "epoch": 0.05710659898477157,
+    "step": 90
+  },
+  {
+    "loss": 0.2111,
+    "grad_norm": 1.2513262033462524,
+    "learning_rate": 6.265822784810128e-05,
+    "epoch": 0.06345177664974619,
+    "step": 100
+  },
+  {
+    "loss": 0.2127,
+    "grad_norm": 1.538354754447937,
+    "learning_rate": 6.89873417721519e-05,
+    "epoch": 0.06979695431472081,
+    "step": 110
+  },
+  {
+    "loss": 0.1974,
+    "grad_norm": 1.3688119649887085,
+    "learning_rate": 7.531645569620254e-05,
+    "epoch": 0.07614213197969544,
+    "step": 120
+  },
+  {
+    "loss": 0.1973,
+    "grad_norm": 1.1792855262756348,
+    "learning_rate": 8.164556962025317e-05,
+    "epoch": 0.08248730964467005,
+    "step": 130
+  },
+  {
+    "loss": 0.1735,
+    "grad_norm": 1.0454416275024414,
+    "learning_rate": 8.797468354430381e-05,
+    "epoch": 0.08883248730964467,
+    "step": 140
+  },
+  {
+    "loss": 0.1464,
+    "grad_norm": 1.3377323150634766,
+    "learning_rate": 9.430379746835444e-05,
+    "epoch": 0.09517766497461928,
+    "step": 150
+  },
+  {
+    "loss": 0.1812,
+    "grad_norm": 1.117480754852295,
+    "learning_rate": 0.00010063291139240508,
+    "epoch": 0.10152284263959391,
+    "step": 160
+  },
+  {
+    "loss": 0.1533,
+    "grad_norm": 1.5500177145004272,
+    "learning_rate": 0.00010696202531645569,
+    "epoch": 0.10786802030456853,
+    "step": 170
+  },
+  {
+    "loss": 0.1348,
+    "grad_norm": 0.986708402633667,
+    "learning_rate": 0.00011329113924050634,
+    "epoch": 0.11421319796954314,
+    "step": 180
+  },
+  {
+    "loss": 0.1342,
+    "grad_norm": 1.2270374298095703,
+    "learning_rate": 0.00011962025316455696,
+    "epoch": 0.12055837563451777,
+    "step": 190
+  },
+  {
+    "loss": 0.1381,
+    "grad_norm": 0.9935520887374878,
+    "learning_rate": 0.0001259493670886076,
+    "epoch": 0.12690355329949238,
+    "step": 200
+  },
+  {
+    "loss": 0.1513,
+    "grad_norm": 1.4393113851547241,
+    "learning_rate": 0.00013227848101265822,
+    "epoch": 0.13324873096446702,
+    "step": 210
+  },
+  {
+    "loss": 0.1221,
+    "grad_norm": 0.9539300799369812,
+    "learning_rate": 0.00013860759493670888,
+    "epoch": 0.13959390862944163,
+    "step": 220
+  },
+  {
+    "loss": 0.1414,
+    "grad_norm": 0.8915525078773499,
+    "learning_rate": 0.0001449367088607595,
+    "epoch": 0.14593908629441624,
+    "step": 230
+  },
+  {
+    "loss": 0.1331,
+    "grad_norm": 0.6808815002441406,
+    "learning_rate": 0.00015126582278481013,
+    "epoch": 0.15228426395939088,
+    "step": 240
+  },
+  {
+    "loss": 0.1371,
+    "grad_norm": 0.7938814759254456,
+    "learning_rate": 0.00015759493670886075,
+    "epoch": 0.15862944162436549,
+    "step": 250
+  },
+  {
+    "loss": 0.1321,
+    "grad_norm": 0.6890779733657837,
+    "learning_rate": 0.0001639240506329114,
+    "epoch": 0.1649746192893401,
+    "step": 260
+  },
+  {
+    "loss": 0.1158,
+    "grad_norm": 0.4467112720012665,
+    "learning_rate": 0.00017025316455696204,
+    "epoch": 0.1713197969543147,
+    "step": 270
+  },
+  {
+    "loss": 0.1334,
+    "grad_norm": 1.0458476543426514,
+    "learning_rate": 0.00017658227848101266,
+    "epoch": 0.17766497461928935,
+    "step": 280
+  },
+  {
+    "loss": 0.1183,
+    "grad_norm": 0.8898298144340515,
+    "learning_rate": 0.0001829113924050633,
+    "epoch": 0.18401015228426396,
+    "step": 290
+  },
+  {
+    "loss": 0.1164,
+    "grad_norm": 0.8841809034347534,
+    "learning_rate": 0.00018924050632911394,
+    "epoch": 0.19035532994923857,
+    "step": 300
+  },
+  {
+    "loss": 0.1178,
+    "grad_norm": 0.7970095276832581,
+    "learning_rate": 0.00019556962025316457,
+    "epoch": 0.1967005076142132,
+    "step": 310
+  },
+  {
+    "eval_loss": 0.11779838055372238,
+    "eval_runtime": 45.5288,
+    "eval_samples_per_second": 29.146,
+    "eval_steps_per_second": 7.292,
+    "epoch": 0.1998730964467005,
+    "step": 315
+  },
+  {
+    "loss": 0.104,
+    "grad_norm": 0.7851632237434387,
+    "learning_rate": 0.00019999944779618028,
+    "epoch": 0.20304568527918782,
+    "step": 320
+  },
+  {
+    "loss": 0.1279,
+    "grad_norm": 0.7609685659408569,
+    "learning_rate": 0.00019998963100903997,
+    "epoch": 0.20939086294416243,
+    "step": 330
+  },
+  {
+    "loss": 0.1225,
+    "grad_norm": 0.764049232006073,
+    "learning_rate": 0.00019996754441248412,
+    "epoch": 0.21573604060913706,
+    "step": 340
+  },
+  {
+    "loss": 0.1057,
+    "grad_norm": 0.7909657955169678,
+    "learning_rate": 0.000199933190716777,
+    "epoch": 0.22208121827411167,
+    "step": 350
+  },
+  {
+    "loss": 0.1036,
+    "grad_norm": 0.9112188816070557,
+    "learning_rate": 0.00019988657413748867,
+    "epoch": 0.22842639593908629,
+    "step": 360
+  },
+  {
+    "loss": 0.1065,
+    "grad_norm": 0.8363418579101562,
+    "learning_rate": 0.0001998277003949778,
+    "epoch": 0.23477157360406092,
+    "step": 370
+  },
+  {
+    "loss": 0.1095,
+    "grad_norm": 1.1866352558135986,
+    "learning_rate": 0.00019975657671368943,
+    "epoch": 0.24111675126903553,
+    "step": 380
+  },
+  {
+    "loss": 0.1049,
+    "grad_norm": 0.5500998497009277,
+    "learning_rate": 0.00019967321182126873,
+    "epoch": 0.24746192893401014,
+    "step": 390
+  },
+  {
+    "loss": 0.1064,
+    "grad_norm": 0.7981701493263245,
+    "learning_rate": 0.00019957761594748975,
+    "epoch": 0.25380710659898476,
+    "step": 400
+  },
+  {
+    "loss": 0.0907,
+    "grad_norm": 0.5598951578140259,
+    "learning_rate": 0.0001994698008230005,
+    "epoch": 0.26015228426395937,
+    "step": 410
+  },
+  {
+    "loss": 0.1064,
+    "grad_norm": 0.8710437417030334,
+    "learning_rate": 0.00019934977967788294,
+    "epoch": 0.26649746192893403,
+    "step": 420
+  },
+  {
+    "loss": 0.107,
+    "grad_norm": 0.5126726627349854,
+    "learning_rate": 0.00019921756724003008,
+    "epoch": 0.27284263959390864,
+    "step": 430
+  },
+  {
+    "loss": 0.0998,
+    "grad_norm": 0.7381883263587952,
+    "learning_rate": 0.0001990731797333383,
+    "epoch": 0.27918781725888325,
+    "step": 440
+  },
+  {
+    "loss": 0.1012,
+    "grad_norm": 0.6682422757148743,
+    "learning_rate": 0.00019891663487571663,
+    "epoch": 0.28553299492385786,
+    "step": 450
+  },
+  {
+    "loss": 0.0988,
+    "grad_norm": 0.5485838651657104,
+    "learning_rate": 0.00019874795187691267,
+    "epoch": 0.2918781725888325,
+    "step": 460
+  },
+  {
+    "loss": 0.0915,
+    "grad_norm": 0.46157270669937134,
+    "learning_rate": 0.00019856715143615512,
+    "epoch": 0.2982233502538071,
+    "step": 470
+  },
+  {
+    "loss": 0.0943,
+    "grad_norm": 0.7231584787368774,
+    "learning_rate": 0.000198374255739614,
+    "epoch": 0.30456852791878175,
+    "step": 480
+  },
+  {
+    "loss": 0.0884,
+    "grad_norm": 0.6452136039733887,
+    "learning_rate": 0.000198169288457678,
+    "epoch": 0.31091370558375636,
+    "step": 490
+  },
+  {
+    "loss": 0.0952,
+    "grad_norm": 0.5756458640098572,
+    "learning_rate": 0.00019795227474205,
+    "epoch": 0.31725888324873097,
+    "step": 500
+  },
+  {
+    "loss": 0.0942,
+    "grad_norm": 0.5166134238243103,
+    "learning_rate": 0.00019772324122266054,
+    "epoch": 0.3236040609137056,
+    "step": 510
+  },
+  {
+    "loss": 0.0981,
+    "grad_norm": 0.7283093333244324,
+    "learning_rate": 0.00019748221600440015,
+    "epoch": 0.3299492385786802,
+    "step": 520
+  },
+  {
+    "loss": 0.0817,
+    "grad_norm": 0.5151678323745728,
+    "learning_rate": 0.00019722922866367054,
+    "epoch": 0.3362944162436548,
+    "step": 530
+  },
+  {
+    "loss": 0.0855,
+    "grad_norm": 0.4918259382247925,
+    "learning_rate": 0.0001969643102447552,
+    "epoch": 0.3426395939086294,
+    "step": 540
+  },
+  {
+    "loss": 0.0902,
+    "grad_norm": 0.4107823669910431,
+    "learning_rate": 0.00019668749325601006,
+    "epoch": 0.3489847715736041,
+    "step": 550
+  },
+  {
+    "loss": 0.0815,
+    "grad_norm": 0.45388150215148926,
+    "learning_rate": 0.00019639881166587427,
+    "epoch": 0.3553299492385787,
+    "step": 560
+  },
+  {
+    "loss": 0.0857,
+    "grad_norm": 0.46063661575317383,
+    "learning_rate": 0.00019609830089870183,
+    "epoch": 0.3616751269035533,
+    "step": 570
+  },
+  {
+    "loss": 0.0892,
+    "grad_norm": 0.43047210574150085,
+    "learning_rate": 0.00019578599783041492,
+    "epoch": 0.3680203045685279,
+    "step": 580
+  },
+  {
+    "loss": 0.0822,
+    "grad_norm": 0.6658169627189636,
+    "learning_rate": 0.0001954619407839784,
+    "epoch": 0.3743654822335025,
+    "step": 590
+  },
+  {
+    "loss": 0.0865,
+    "grad_norm": 0.686517596244812,
+    "learning_rate": 0.0001951261695246976,
+    "epoch": 0.38071065989847713,
+    "step": 600
+  },
+  {
+    "loss": 0.0843,
+    "grad_norm": 0.39768409729003906,
+    "learning_rate": 0.00019477872525533845,
+    "epoch": 0.3870558375634518,
+    "step": 610
+  },
+  {
+    "loss": 0.0921,
+    "grad_norm": 0.5196660757064819,
+    "learning_rate": 0.0001944196506110714,
+    "epoch": 0.3934010152284264,
+    "step": 620
+  },
+  {
+    "loss": 0.0957,
+    "grad_norm": 0.5350348949432373,
+    "learning_rate": 0.0001940489896542398,
+    "epoch": 0.399746192893401,
+    "step": 630
+  },
+  {
+    "eval_loss": 0.08027429133653641,
+    "eval_runtime": 23.4642,
+    "eval_samples_per_second": 56.554,
+    "eval_steps_per_second": 14.149,
+    "epoch": 0.399746192893401,
+    "step": 630
+  },
+  {
+    "loss": 0.0911,
+    "grad_norm": 0.7363661527633667,
+    "learning_rate": 0.00019366678786895286,
+    "epoch": 0.40609137055837563,
+    "step": 640
+  },
+  {
+    "loss": 0.0789,
+    "grad_norm": 0.3176012933254242,
+    "learning_rate": 0.00019327309215550433,
+    "epoch": 0.41243654822335024,
+    "step": 650
+  },
+  {
+    "loss": 0.0875,
+    "grad_norm": 0.4339086711406708,
+    "learning_rate": 0.00019286795082461722,
+    "epoch": 0.41878172588832485,
+    "step": 660
+  },
+  {
+    "loss": 0.0934,
+    "grad_norm": 0.5191656351089478,
+    "learning_rate": 0.00019245141359151572,
+    "epoch": 0.4251269035532995,
+    "step": 670
+  },
+  {
+    "loss": 0.0825,
+    "grad_norm": 0.41910088062286377,
+    "learning_rate": 0.0001920235315698244,
+    "epoch": 0.43147208121827413,
+    "step": 680
+  },
+  {
+    "loss": 0.0823,
+    "grad_norm": 0.7931292057037354,
+    "learning_rate": 0.0001915843572652962,
+    "epoch": 0.43781725888324874,
+    "step": 690
+  },
+  {
+    "loss": 0.0931,
+    "grad_norm": 0.6371334195137024,
+    "learning_rate": 0.00019113394456936927,
+    "epoch": 0.44416243654822335,
+    "step": 700
+  },
+  {
+    "loss": 0.0844,
+    "grad_norm": 0.46574267745018005,
+    "learning_rate": 0.00019067234875255393,
+    "epoch": 0.45050761421319796,
+    "step": 710
+  },
+  {
+    "loss": 0.0893,
+    "grad_norm": 0.5641367435455322,
+    "learning_rate": 0.00019019962645765054,
+    "epoch": 0.45685279187817257,
+    "step": 720
+  },
+  {
+    "loss": 0.0758,
+    "grad_norm": 0.4197913408279419,
+    "learning_rate": 0.0001897158356927985,
+    "epoch": 0.4631979695431472,
+    "step": 730
+  },
+  {
+    "loss": 0.0748,
+    "grad_norm": 0.5938242673873901,
+    "learning_rate": 0.00018922103582435845,
+    "epoch": 0.46954314720812185,
+    "step": 740
+  },
+  {
+    "loss": 0.082,
+    "grad_norm": 0.4360695481300354,
+    "learning_rate": 0.00018871528756962696,
+    "epoch": 0.47588832487309646,
+    "step": 750
+  },
+  {
+    "loss": 0.0767,
+    "grad_norm": 0.4777214229106903,
+    "learning_rate": 0.00018819865298938605,
+    "epoch": 0.48223350253807107,
+    "step": 760
+  },
+  {
+    "loss": 0.0681,
+    "grad_norm": 0.7168831825256348,
+    "learning_rate": 0.0001876711954802877,
+    "epoch": 0.4885786802030457,
+    "step": 770
+  },
+  {
+    "loss": 0.0754,
+    "grad_norm": 0.46045243740081787,
+    "learning_rate": 0.00018713297976707437,
+    "epoch": 0.4949238578680203,
+    "step": 780
+  },
+  {
+    "loss": 0.0853,
+    "grad_norm": 0.6867933869361877,
+    "learning_rate": 0.00018658407189463643,
+    "epoch": 0.501269035532995,
+    "step": 790
+  },
+  {
+    "loss": 0.0831,
+    "grad_norm": 0.3895761966705322,
+    "learning_rate": 0.00018602453921990798,
+    "epoch": 0.5076142131979695,
+    "step": 800
+  },
+  {
+    "loss": 0.081,
+    "grad_norm": 0.4646884500980377,
+    "learning_rate": 0.00018545445040360123,
+    "epoch": 0.5139593908629442,
+    "step": 810
+  },
+  {
+    "loss": 0.0729,
+    "grad_norm": 0.4897511601448059,
+    "learning_rate": 0.00018487387540178118,
+    "epoch": 0.5203045685279187,
+    "step": 820
+  },
+  {
+    "loss": 0.0849,
+    "grad_norm": 0.5871896743774414,
+    "learning_rate": 0.00018428288545728125,
+    "epoch": 0.5266497461928934,
+    "step": 830
+  },
+  {
+    "loss": 0.0751,
+    "grad_norm": 0.4862145781517029,
+    "learning_rate": 0.000183681553090961,
+    "epoch": 0.5329949238578681,
+    "step": 840
+  },
+  {
+    "loss": 0.0643,
+    "grad_norm": 0.4235672950744629,
+    "learning_rate": 0.0001830699520928069,
+    "epoch": 0.5393401015228426,
+    "step": 850
+  },
+  {
+    "loss": 0.075,
+    "grad_norm": 0.28317147493362427,
+    "learning_rate": 0.00018244815751287786,
+    "epoch": 0.5456852791878173,
+    "step": 860
+  },
+  {
+    "loss": 0.0731,
+    "grad_norm": 0.4897977411746979,
+    "learning_rate": 0.00018181624565209535,
+    "epoch": 0.5520304568527918,
+    "step": 870
+  },
+  {
+    "loss": 0.09,
+    "grad_norm": 0.4528476595878601,
+    "learning_rate": 0.0001811742940528808,
+    "epoch": 0.5583756345177665,
+    "step": 880
+  },
+  {
+    "loss": 0.0906,
+    "grad_norm": 0.4155082106590271,
+    "learning_rate": 0.00018052238148964005,
+    "epoch": 0.5647208121827412,
+    "step": 890
+  },
+  {
+    "loss": 0.0794,
+    "grad_norm": 0.525433361530304,
+    "learning_rate": 0.00017986058795909702,
+    "epoch": 0.5710659898477157,
+    "step": 900
+  },
+  {
+    "loss": 0.0708,
+    "grad_norm": 0.4712257981300354,
+    "learning_rate": 0.00017918899467047716,
+    "epoch": 0.5774111675126904,
+    "step": 910
+  },
+  {
+    "loss": 0.0758,
+    "grad_norm": 0.4515252709388733,
+    "learning_rate": 0.00017850768403554238,
+    "epoch": 0.583756345177665,
+    "step": 920
+  },
+  {
+    "loss": 0.0773,
+    "grad_norm": 0.5485005378723145,
+    "learning_rate": 0.00017781673965847801,
+    "epoch": 0.5901015228426396,
+    "step": 930
+  },
+  {
+    "loss": 0.0759,
+    "grad_norm": 0.5751654505729675,
+    "learning_rate": 0.00017711624632563372,
+    "epoch": 0.5964467005076142,
+    "step": 940
+  },
+  {
+    "eval_loss": 0.07487394660711288,
+    "eval_runtime": 23.4569,
+    "eval_samples_per_second": 56.572,
+    "eval_steps_per_second": 14.154,
+    "epoch": 0.5996192893401016,
+    "step": 945
+  },
+  {
+    "loss": 0.0849,
+    "grad_norm": 0.44699952006340027,
+    "learning_rate": 0.00017640628999511946,
+    "epoch": 0.6027918781725888,
+    "step": 950
+  },
+  {
+    "loss": 0.0653,
+    "grad_norm": 0.2837465703487396,
+    "learning_rate": 0.00017568695778625727,
+    "epoch": 0.6091370558375635,
+    "step": 960
+  },
+  {
+    "loss": 0.0764,
+    "grad_norm": 0.3115606904029846,
+    "learning_rate": 0.00017495833796889098,
+    "epoch": 0.6154822335025381,
+    "step": 970
+  },
+  {
+    "loss": 0.0796,
+    "grad_norm": 0.7619908452033997,
+    "learning_rate": 0.0001742205199525544,
+    "epoch": 0.6218274111675127,
+    "step": 980
+  },
+  {
+    "loss": 0.0748,
+    "grad_norm": 0.7126716375350952,
+    "learning_rate": 0.00017347359427549982,
+    "epoch": 0.6281725888324873,
+    "step": 990
+  },
+  {
+    "loss": 0.0828,
+    "grad_norm": 0.42106711864471436,
+    "learning_rate": 0.00017271765259358816,
+    "epoch": 0.6345177664974619,
+    "step": 1000
+  },
+  {
+    "loss": 0.0755,
+    "grad_norm": 0.7738011479377747,
+    "learning_rate": 0.00017195278766904157,
+    "epoch": 0.6408629441624365,
+    "step": 1010
+  },
+  {
+    "loss": 0.0783,
+    "grad_norm": 0.4499462842941284,
+    "learning_rate": 0.00017117909335906056,
+    "epoch": 0.6472081218274112,
+    "step": 1020
+  },
+  {
+    "loss": 0.0868,
+    "grad_norm": 0.5416187047958374,
+    "learning_rate": 0.00017039666460430688,
+    "epoch": 0.6535532994923858,
+    "step": 1030
+  },
+  {
+    "loss": 0.0601,
+    "grad_norm": 0.5818226933479309,
+    "learning_rate": 0.00016960559741725304,
+    "epoch": 0.6598984771573604,
+    "step": 1040
+  },
+  {
+    "loss": 0.0789,
+    "grad_norm": 0.5406737923622131,
+    "learning_rate": 0.0001688059888704007,
+    "epoch": 0.666243654822335,
+    "step": 1050
+  },
+  {
+    "loss": 0.076,
+    "grad_norm": 0.449704110622406,
+    "learning_rate": 0.00016799793708436873,
+    "epoch": 0.6725888324873096,
+    "step": 1060
+  },
+  {
+    "loss": 0.0796,
+    "grad_norm": 0.3174469769001007,
+    "learning_rate": 0.00016718154121585284,
+    "epoch": 0.6789340101522843,
+    "step": 1070
+  },
+  {
+    "loss": 0.0771,
+    "grad_norm": 0.5006186962127686,
+    "learning_rate": 0.00016635690144545782,
+    "epoch": 0.6852791878172588,
+    "step": 1080
+  },
+  {
+    "loss": 0.0652,
+    "grad_norm": 0.45733827352523804,
+    "learning_rate": 0.0001655241189654045,
+    "epoch": 0.6916243654822335,
+    "step": 1090
+  },
+  {
+    "loss": 0.068,
+    "grad_norm": 0.4655713140964508,
+    "learning_rate": 0.00016468329596711233,
+    "epoch": 0.6979695431472082,
+    "step": 1100
+  },
+  {
+    "loss": 0.0819,
+    "grad_norm": 0.5244554281234741,
+    "learning_rate": 0.0001638345356286592,
+    "epoch": 0.7043147208121827,
+    "step": 1110
+  },
+  {
+    "loss": 0.071,
+    "grad_norm": 0.3699105381965637,
+    "learning_rate": 0.00016297794210212049,
+    "epoch": 0.7106598984771574,
+    "step": 1120
+  },
+  {
+    "loss": 0.069,
+    "grad_norm": 0.2289179265499115,
+    "learning_rate": 0.00016211362050078863,
+    "epoch": 0.7170050761421319,
+    "step": 1130
+  },
+  {
+    "loss": 0.0748,
+    "grad_norm": 0.43648213148117065,
+    "learning_rate": 0.00016124167688627434,
+    "epoch": 0.7233502538071066,
+    "step": 1140
+  },
+  {
+    "loss": 0.0646,
+    "grad_norm": 0.42901450395584106,
+    "learning_rate": 0.00016036221825549185,
+    "epoch": 0.7296954314720813,
+    "step": 1150
+  },
+  {
+    "loss": 0.067,
+    "grad_norm": 0.3802100121974945,
+    "learning_rate": 0.00015947535252752913,
+    "epoch": 0.7360406091370558,
+    "step": 1160
+  },
+  {
+    "loss": 0.0786,
+    "grad_norm": 0.49444833397865295,
+    "learning_rate": 0.00015858118853040516,
+    "epoch": 0.7423857868020305,
+    "step": 1170
+  },
+  {
+    "loss": 0.0741,
+    "grad_norm": 0.4874003231525421,
+    "learning_rate": 0.00015767983598771545,
+    "epoch": 0.748730964467005,
+    "step": 1180
+  },
+  {
+    "loss": 0.0691,
+    "grad_norm": 0.3816661238670349,
+    "learning_rate": 0.0001567714055051679,
+    "epoch": 0.7550761421319797,
+    "step": 1190
+  },
+  {
+    "loss": 0.0693,
+    "grad_norm": 0.3660362660884857,
+    "learning_rate": 0.00015585600855701011,
+    "epoch": 0.7614213197969543,
+    "step": 1200
+  },
+  {
+    "loss": 0.0718,
+    "grad_norm": 0.4611922800540924,
+    "learning_rate": 0.00015493375747235047,
+    "epoch": 0.7677664974619289,
+    "step": 1210
+  },
+  {
+    "loss": 0.0753,
+    "grad_norm": 0.3931231200695038,
+    "learning_rate": 0.00015400476542137392,
+    "epoch": 0.7741116751269036,
+    "step": 1220
+  },
+  {
+    "loss": 0.0725,
+    "grad_norm": 0.7242083549499512,
+    "learning_rate": 0.00015306914640145504,
+    "epoch": 0.7804568527918782,
+    "step": 1230
+  },
+  {
+    "loss": 0.0826,
+    "grad_norm": 0.36801645159721375,
+    "learning_rate": 0.0001521270152231691,
+    "epoch": 0.7868020304568528,
+    "step": 1240
+  },
+  {
+    "loss": 0.072,
+    "grad_norm": 0.4194789528846741,
+    "learning_rate": 0.00015117848749620366,
+    "epoch": 0.7931472081218274,
+    "step": 1250
+  },
+  {
+    "loss": 0.0663,
+    "grad_norm": 0.5587615370750427,
+    "learning_rate": 0.00015022367961517208,
+    "epoch": 0.799492385786802,
+    "step": 1260
+  },
+  {
+    "eval_loss": 0.06896253675222397,
+    "eval_runtime": 23.7164,
+    "eval_samples_per_second": 55.953,
+    "eval_steps_per_second": 13.999,
+    "epoch": 0.799492385786802,
+    "step": 1260
+  },
+  {
+    "loss": 0.0678,
+    "grad_norm": 0.26218849420547485,
+    "learning_rate": 0.00014926270874533043,
+    "epoch": 0.8058375634517766,
+    "step": 1270
+  },
+  {
+    "loss": 0.0711,
+    "grad_norm": 0.3901388943195343,
+    "learning_rate": 0.0001482956928082003,
+    "epoch": 0.8121827411167513,
+    "step": 1280
+  },
+  {
+    "loss": 0.0597,
+    "grad_norm": 0.4186069965362549,
+    "learning_rate": 0.0001473227504670984,
+    "epoch": 0.8185279187817259,
+    "step": 1290
+  },
+  {
+    "loss": 0.0624,
+    "grad_norm": 0.3466233015060425,
+    "learning_rate": 0.00014634400111257528,
+    "epoch": 0.8248730964467005,
+    "step": 1300
+  },
+  {
+    "loss": 0.0643,
+    "grad_norm": 0.7102236151695251,
+    "learning_rate": 0.00014535956484776482,
+    "epoch": 0.8312182741116751,
+    "step": 1310
+  },
+  {
+    "loss": 0.0687,
+    "grad_norm": 0.44185611605644226,
+    "learning_rate": 0.00014436956247364645,
+    "epoch": 0.8375634517766497,
+    "step": 1320
+  },
+  {
+    "loss": 0.0676,
+    "grad_norm": 0.4165748953819275,
+    "learning_rate": 0.00014337411547422132,
+    "epoch": 0.8439086294416244,
+    "step": 1330
+  },
+  {
+    "loss": 0.0767,
+    "grad_norm": 0.45543935894966125,
+    "learning_rate": 0.00014237334600160505,
+    "epoch": 0.850253807106599,
+    "step": 1340
+  },
+  {
+    "loss": 0.0677,
+    "grad_norm": 0.3414968252182007,
+    "learning_rate": 0.00014136737686103827,
+    "epoch": 0.8565989847715736,
+    "step": 1350
+  },
+  {
+    "loss": 0.0604,
+    "grad_norm": 0.2950080931186676,
+    "learning_rate": 0.00014035633149581716,
+    "epoch": 0.8629441624365483,
+    "step": 1360
+  },
+  {
+    "loss": 0.0767,
+    "grad_norm": 0.3940573036670685,
+    "learning_rate": 0.0001393403339721455,
+    "epoch": 0.8692893401015228,
+    "step": 1370
+  },
+  {
+    "loss": 0.0634,
+    "grad_norm": 0.39822471141815186,
+    "learning_rate": 0.00013831950896391054,
+    "epoch": 0.8756345177664975,
+    "step": 1380
+  },
+  {
+    "loss": 0.065,
+    "grad_norm": 0.4311733543872833,
+    "learning_rate": 0.00013729398173738415,
+    "epoch": 0.881979695431472,
+    "step": 1390
+  },
+  {
+    "loss": 0.0555,
+    "grad_norm": 0.37173038721084595,
+    "learning_rate": 0.00013626387813585116,
+    "epoch": 0.8883248730964467,
+    "step": 1400
+  },
+  {
+    "loss": 0.0672,
+    "grad_norm": 0.419643759727478,
+    "learning_rate": 0.0001352293245641672,
+    "epoch": 0.8946700507614214,
+    "step": 1410
+  },
+  {
+    "loss": 0.0746,
+    "grad_norm": 0.29487496614456177,
+    "learning_rate": 0.00013419044797324738,
+    "epoch": 0.9010152284263959,
+    "step": 1420
+  },
+  {
+    "loss": 0.0715,
+    "grad_norm": 0.536567747592926,
+    "learning_rate": 0.00013314737584448796,
+    "epoch": 0.9073604060913706,
+    "step": 1430
+  },
+  {
+    "loss": 0.0681,
+    "grad_norm": 0.3137727677822113,
+    "learning_rate": 0.00013210023617412313,
+    "epoch": 0.9137055837563451,
+    "step": 1440
+  },
+  {
+    "loss": 0.0754,
+    "grad_norm": 0.25659045577049255,
+    "learning_rate": 0.0001310491574575184,
+    "epoch": 0.9200507614213198,
+    "step": 1450
+  },
+  {
+    "loss": 0.0654,
+    "grad_norm": 0.6076952219009399,
+    "learning_rate": 0.00012999426867340295,
+    "epoch": 0.9263959390862944,
+    "step": 1460
+  },
+  {
+    "loss": 0.0619,
+    "grad_norm": 0.3243611752986908,
+    "learning_rate": 0.00012893569926804237,
+    "epoch": 0.932741116751269,
+    "step": 1470
+  },
+  {
+    "loss": 0.0567,
+    "grad_norm": 0.5209382772445679,
+    "learning_rate": 0.00012787357913935437,
+    "epoch": 0.9390862944162437,
+    "step": 1480
+  },
+  {
+    "loss": 0.0631,
+    "grad_norm": 0.36193498969078064,
+    "learning_rate": 0.00012680803862096874,
+    "epoch": 0.9454314720812182,
+    "step": 1490
+  },
+  {
+    "loss": 0.0609,
+    "grad_norm": 0.3028424382209778,
+    "learning_rate": 0.0001257392084662342,
+    "epoch": 0.9517766497461929,
+    "step": 1500
+  },
+  {
+    "loss": 0.0721,
+    "grad_norm": 0.6666154861450195,
+    "learning_rate": 0.00012466721983217332,
+    "epoch": 0.9581218274111675,
+    "step": 1510
+  },
+  {
+    "loss": 0.0603,
+    "grad_norm": 0.3871685564517975,
+    "learning_rate": 0.0001235922042633883,
+    "epoch": 0.9644670050761421,
+    "step": 1520
+  },
+  {
+    "loss": 0.0644,
+    "grad_norm": 0.25319841504096985,
+    "learning_rate": 0.00012251429367591903,
+    "epoch": 0.9708121827411168,
+    "step": 1530
+  },
+  {
+    "loss": 0.0582,
+    "grad_norm": 0.34290412068367004,
+    "learning_rate": 0.00012143362034105541,
+    "epoch": 0.9771573604060914,
+    "step": 1540
+  },
+  {
+    "loss": 0.0612,
+    "grad_norm": 0.39071834087371826,
+    "learning_rate": 0.0001203503168691064,
+    "epoch": 0.983502538071066,
+    "step": 1550
+  },
+  {
+    "loss": 0.0609,
+    "grad_norm": 0.3652092516422272,
+    "learning_rate": 0.00011926451619312724,
+    "epoch": 0.9898477157360406,
+    "step": 1560
+  },
+  {
+    "loss": 0.0721,
+    "grad_norm": 0.29865705966949463,
+    "learning_rate": 0.00011817635155260707,
+    "epoch": 0.9961928934010152,
+    "step": 1570
+  },
+  {
+    "eval_loss": 0.061239708214998245,
+    "eval_runtime": 23.5194,
+    "eval_samples_per_second": 56.421,
+    "eval_steps_per_second": 14.116,
+    "epoch": 0.9993654822335025,
+    "step": 1575
+  },
+  {
+    "loss": 0.0582,
+    "grad_norm": 0.2990482449531555,
+    "learning_rate": 0.00011708595647711909,
+    "epoch": 1.00253807106599,
+    "step": 1580
+  },
+  {
+    "loss": 0.0416,
+    "grad_norm": 0.3069770932197571,
+    "learning_rate": 0.00011599346476993495,
+    "epoch": 1.0088832487309645,
+    "step": 1590
+  },
+  {
+    "loss": 0.0434,
+    "grad_norm": 0.29351532459259033,
+    "learning_rate": 0.0001148990104916058,
+    "epoch": 1.015228426395939,
+    "step": 1600
+  },
+  {
+    "loss": 0.0359,
+    "grad_norm": 0.2513970136642456,
+    "learning_rate": 0.00011380272794351138,
+    "epoch": 1.0215736040609138,
+    "step": 1610
+  },
+  {
+    "loss": 0.0542,
+    "grad_norm": 0.3737475574016571,
+    "learning_rate": 0.00011270475165138,
+    "epoch": 1.0279187817258884,
+    "step": 1620
+  },
+  {
+    "loss": 0.0444,
+    "grad_norm": 0.43037623167037964,
+    "learning_rate": 0.00011160521634878058,
+    "epoch": 1.034263959390863,
+    "step": 1630
+  },
+  {
+    "loss": 0.0393,
+    "grad_norm": 0.5113381147384644,
+    "learning_rate": 0.00011050425696058963,
+    "epoch": 1.0406091370558375,
+    "step": 1640
+  },
+  {
+    "loss": 0.0447,
+    "grad_norm": 0.40034595131874084,
+    "learning_rate": 0.00010940200858643424,
+    "epoch": 1.0469543147208122,
+    "step": 1650
+  },
+  {
+    "loss": 0.0378,
+    "grad_norm": 0.3174040615558624,
+    "learning_rate": 0.00010829860648411407,
+    "epoch": 1.0532994923857868,
+    "step": 1660
+  },
+  {
+    "loss": 0.0391,
+    "grad_norm": 0.4041596055030823,
+    "learning_rate": 0.00010719418605300374,
+    "epoch": 1.0596446700507614,
+    "step": 1670
+  },
+  {
+    "loss": 0.0379,
+    "grad_norm": 0.3625265061855316,
+    "learning_rate": 0.00010608888281743778,
+    "epoch": 1.0659898477157361,
+    "step": 1680
+  },
+  {
+    "loss": 0.031,
+    "grad_norm": 0.4304555058479309,
+    "learning_rate": 0.0001049828324100804,
+    "epoch": 1.0723350253807107,
+    "step": 1690
+  },
+  {
+    "loss": 0.0487,
+    "grad_norm": 0.2591390311717987,
+    "learning_rate": 0.0001038761705552819,
+    "epoch": 1.0786802030456852,
+    "step": 1700
+  },
+  {
+    "loss": 0.0394,
+    "grad_norm": 0.3746626377105713,
+    "learning_rate": 0.00010276903305242372,
+    "epoch": 1.0850253807106598,
+    "step": 1710
+  },
+  {
+    "loss": 0.0411,
+    "grad_norm": 0.3750106990337372,
+    "learning_rate": 0.00010166155575925454,
+    "epoch": 1.0913705583756346,
+    "step": 1720
+  },
+  {
+    "loss": 0.0373,
+    "grad_norm": 0.45232951641082764,
+    "learning_rate": 0.00010055387457521892,
+    "epoch": 1.0977157360406091,
+    "step": 1730
+  },
+  {
+    "loss": 0.0409,
+    "grad_norm": 0.40455761551856995,
+    "learning_rate": 9.944612542478109e-05,
+    "epoch": 1.1040609137055837,
+    "step": 1740
+  },
+  {
+    "loss": 0.0404,
+    "grad_norm": 0.45171070098876953,
+    "learning_rate": 9.833844424074546e-05,
+    "epoch": 1.1104060913705585,
+    "step": 1750
+  },
+  {
+    "loss": 0.0354,
+    "grad_norm": 0.16944268345832825,
+    "learning_rate": 9.723096694757629e-05,
+    "epoch": 1.116751269035533,
+    "step": 1760
+  },
+  {
+    "loss": 0.0449,
+    "grad_norm": 0.3960643410682678,
+    "learning_rate": 9.612382944471811e-05,
+    "epoch": 1.1230964467005076,
+    "step": 1770
+  },
+  {
+    "loss": 0.0383,
+    "grad_norm": 0.39104658365249634,
+    "learning_rate": 9.50171675899196e-05,
+    "epoch": 1.1294416243654823,
+    "step": 1780
+  },
+  {
+    "loss": 0.0401,
+    "grad_norm": 0.3628897964954376,
+    "learning_rate": 9.391111718256225e-05,
+    "epoch": 1.135786802030457,
+    "step": 1790
+  },
+  {
+    "loss": 0.038,
+    "grad_norm": 0.46199262142181396,
+    "learning_rate": 9.28058139469963e-05,
+    "epoch": 1.1421319796954315,
+    "step": 1800
+  },
+  {
+    "loss": 0.0404,
+    "grad_norm": 0.5942744612693787,
+    "learning_rate": 9.170139351588594e-05,
+    "epoch": 1.148477157360406,
+    "step": 1810
+  },
+  {
+    "loss": 0.0359,
+    "grad_norm": 0.46782055497169495,
+    "learning_rate": 9.059799141356579e-05,
+    "epoch": 1.1548223350253808,
+    "step": 1820
+  },
+  {
+    "loss": 0.0413,
+    "grad_norm": 0.42671218514442444,
+    "learning_rate": 8.949574303941039e-05,
+    "epoch": 1.1611675126903553,
+    "step": 1830
+  },
+  {
+    "loss": 0.0455,
+    "grad_norm": 0.41504427790641785,
+    "learning_rate": 8.839478365121943e-05,
+    "epoch": 1.16751269035533,
+    "step": 1840
+  },
+  {
+    "loss": 0.0446,
+    "grad_norm": 0.19736254215240479,
+    "learning_rate": 8.729524834862003e-05,
+    "epoch": 1.1738578680203045,
+    "step": 1850
+  },
+  {
+    "loss": 0.0435,
+    "grad_norm": 0.4627068340778351,
+    "learning_rate": 8.619727205648863e-05,
+    "epoch": 1.1802030456852792,
+    "step": 1860
+  },
+  {
+    "loss": 0.0411,
+    "grad_norm": 0.4953509569168091,
+    "learning_rate": 8.51009895083942e-05,
+    "epoch": 1.1865482233502538,
+    "step": 1870
+  },
+  {
+    "loss": 0.0311,
+    "grad_norm": 0.21544459462165833,
+    "learning_rate": 8.400653523006506e-05,
+    "epoch": 1.1928934010152283,
+    "step": 1880
+  },
+  {
+    "loss": 0.0469,
+    "grad_norm": 0.4519999921321869,
+    "learning_rate": 8.291404352288095e-05,
+    "epoch": 1.1992385786802031,
+    "step": 1890
+  },
+  {
+    "eval_loss": 0.05870498716831207,
+    "eval_runtime": 23.3902,
+    "eval_samples_per_second": 56.733,
+    "eval_steps_per_second": 14.194,
+    "epoch": 1.1992385786802031,
+    "step": 1890
+  },
+  {
+    "loss": 0.0466,
+    "grad_norm": 0.6195783019065857,
+    "learning_rate": 8.182364844739297e-05,
+    "epoch": 1.2055837563451777,
+    "step": 1900
+  },
+  {
+    "loss": 0.0363,
+    "grad_norm": 0.4518129229545593,
+    "learning_rate": 8.07354838068728e-05,
+    "epoch": 1.2119289340101522,
+    "step": 1910
+  },
+  {
+    "loss": 0.0455,
+    "grad_norm": 0.47803568840026855,
+    "learning_rate": 7.964968313089363e-05,
+    "epoch": 1.218274111675127,
+    "step": 1920
+  },
+  {
+    "loss": 0.0395,
+    "grad_norm": 0.17739883065223694,
+    "learning_rate": 7.856637965894462e-05,
+    "epoch": 1.2246192893401016,
+    "step": 1930
+  },
+  {
+    "loss": 0.0415,
+    "grad_norm": 0.3920941650867462,
+    "learning_rate": 7.748570632408099e-05,
+    "epoch": 1.2309644670050761,
+    "step": 1940
+  },
+  {
+    "loss": 0.0377,
+    "grad_norm": 0.3294707238674164,
+    "learning_rate": 7.640779573661171e-05,
+    "epoch": 1.2373096446700507,
+    "step": 1950
+  },
+  {
+    "loss": 0.039,
+    "grad_norm": 0.34252774715423584,
+    "learning_rate": 7.53327801678267e-05,
+    "epoch": 1.2436548223350254,
+    "step": 1960
+  },
+  {
+    "loss": 0.0403,
+    "grad_norm": 0.4406156837940216,
+    "learning_rate": 7.426079153376582e-05,
+    "epoch": 1.25,
+    "step": 1970
+  },
+  {
+    "loss": 0.0381,
+    "grad_norm": 0.33009278774261475,
+    "learning_rate": 7.319196137903126e-05,
+    "epoch": 1.2563451776649746,
+    "step": 1980
+  },
+  {
+    "loss": 0.0397,
+    "grad_norm": 0.4772908389568329,
+    "learning_rate": 7.212642086064564e-05,
+    "epoch": 1.262690355329949,
+    "step": 1990
+  },
+  {
+    "loss": 0.0433,
+    "grad_norm": 0.3636586666107178,
+    "learning_rate": 7.106430073195764e-05,
+    "epoch": 1.2690355329949239,
+    "step": 2000
+  },
+  {
+    "loss": 0.0399,
+    "grad_norm": 0.37633877992630005,
+    "learning_rate": 7.000573132659706e-05,
+    "epoch": 1.2753807106598984,
+    "step": 2010
+  },
+  {
+    "loss": 0.0425,
+    "grad_norm": 0.3575541377067566,
+    "learning_rate": 6.895084254248162e-05,
+    "epoch": 1.281725888324873,
+    "step": 2020
+  },
+  {
+    "loss": 0.0475,
+    "grad_norm": 0.41141068935394287,
+    "learning_rate": 6.78997638258769e-05,
+    "epoch": 1.2880710659898478,
+    "step": 2030
+  },
+  {
+    "loss": 0.0391,
+    "grad_norm": 0.26739487051963806,
+    "learning_rate": 6.685262415551207e-05,
+    "epoch": 1.2944162436548223,
+    "step": 2040
+  },
+  {
+    "loss": 0.0432,
+    "grad_norm": 0.40416017174720764,
+    "learning_rate": 6.580955202675263e-05,
+    "epoch": 1.3007614213197969,
+    "step": 2050
+  },
+  {
+    "loss": 0.0428,
+    "grad_norm": 0.46243521571159363,
+    "learning_rate": 6.47706754358328e-05,
+    "epoch": 1.3071065989847717,
+    "step": 2060
+  },
+  {
+    "loss": 0.036,
+    "grad_norm": 0.4293893575668335,
+    "learning_rate": 6.373612186414883e-05,
+    "epoch": 1.3134517766497462,
+    "step": 2070
+  },
+  {
+    "loss": 0.0392,
+    "grad_norm": 0.29024600982666016,
+    "learning_rate": 6.270601826261589e-05,
+    "epoch": 1.3197969543147208,
+    "step": 2080
+  },
+  {
+    "loss": 0.0376,
+    "grad_norm": 0.4428802728652954,
+    "learning_rate": 6.168049103608948e-05,
+    "epoch": 1.3261421319796955,
+    "step": 2090
+  },
+  {
+    "loss": 0.038,
+    "grad_norm": 0.38020622730255127,
+    "learning_rate": 6.0659666027854555e-05,
+    "epoch": 1.33248730964467,
+    "step": 2100
+  },
+  {
+    "loss": 0.0327,
+    "grad_norm": 0.3990342915058136,
+    "learning_rate": 5.96436685041829e-05,
+    "epoch": 1.3388324873096447,
+    "step": 2110
+  },
+  {
+    "loss": 0.0382,
+    "grad_norm": 0.3502098023891449,
+    "learning_rate": 5.863262313896176e-05,
+    "epoch": 1.3451776649746192,
+    "step": 2120
+  },
+  {
+    "loss": 0.0359,
+    "grad_norm": 0.1891971081495285,
+    "learning_rate": 5.7626653998394984e-05,
+    "epoch": 1.351522842639594,
+    "step": 2130
+  },
+  {
+    "loss": 0.0573,
+    "grad_norm": 0.5771194100379944,
+    "learning_rate": 5.6625884525778706e-05,
+    "epoch": 1.3578680203045685,
+    "step": 2140
+  },
+  {
+    "loss": 0.043,
+    "grad_norm": 0.38253533840179443,
+    "learning_rate": 5.5630437526353554e-05,
+    "epoch": 1.364213197969543,
+    "step": 2150
+  },
+  {
+    "loss": 0.0462,
+    "grad_norm": 0.3203031122684479,
+    "learning_rate": 5.464043515223518e-05,
+    "epoch": 1.3705583756345177,
+    "step": 2160
+  },
+  {
+    "loss": 0.0339,
+    "grad_norm": 0.259575754404068,
+    "learning_rate": 5.365599888742476e-05,
+    "epoch": 1.3769035532994924,
+    "step": 2170
+  },
+  {
+    "loss": 0.039,
+    "grad_norm": 0.2003287672996521,
+    "learning_rate": 5.267724953290161e-05,
+    "epoch": 1.383248730964467,
+    "step": 2180
+  },
+  {
+    "loss": 0.0347,
+    "grad_norm": 0.38988855481147766,
+    "learning_rate": 5.170430719179966e-05,
+    "epoch": 1.3895939086294415,
+    "step": 2190
+  },
+  {
+    "loss": 0.0363,
+    "grad_norm": 0.29437464475631714,
+    "learning_rate": 5.073729125466957e-05,
+    "epoch": 1.3959390862944163,
+    "step": 2200
+  },
+  {
+    "eval_loss": 0.057627856731414795,
+    "eval_runtime": 23.9115,
+    "eval_samples_per_second": 55.496,
+    "eval_steps_per_second": 13.885,
+    "epoch": 1.3991116751269035,
+    "step": 2205
+  },
+  {
+    "loss": 0.0359,
+    "grad_norm": 0.4737837314605713,
+    "learning_rate": 4.9776320384827926e-05,
+    "epoch": 1.4022842639593909,
+    "step": 2210
+  },
+  {
+    "loss": 0.0332,
+    "grad_norm": 0.3594450056552887,
+    "learning_rate": 4.882151250379635e-05,
+    "epoch": 1.4086294416243654,
+    "step": 2220
+  },
+  {
+    "loss": 0.0425,
+    "grad_norm": 0.40817004442214966,
+    "learning_rate": 4.787298477683092e-05,
+    "epoch": 1.4149746192893402,
+    "step": 2230
+  },
+  {
+    "loss": 0.0364,
+    "grad_norm": 0.43428879976272583,
+    "learning_rate": 4.6930853598544997e-05,
+    "epoch": 1.4213197969543148,
+    "step": 2240
+  },
+  {
+    "loss": 0.0357,
+    "grad_norm": 0.40719398856163025,
+    "learning_rate": 4.599523457862609e-05,
+    "epoch": 1.4276649746192893,
+    "step": 2250
+  },
+  {
+    "loss": 0.0424,
+    "grad_norm": 0.3827572762966156,
+    "learning_rate": 4.506624252764954e-05,
+    "epoch": 1.434010152284264,
+    "step": 2260
+  },
+  {
+    "loss": 0.0393,
+    "grad_norm": 0.3325338065624237,
+    "learning_rate": 4.414399144298989e-05,
+    "epoch": 1.4403553299492386,
+    "step": 2270
+  },
+  {
+    "loss": 0.0406,
+    "grad_norm": 0.1980651319026947,
+    "learning_rate": 4.3228594494832106e-05,
+    "epoch": 1.4467005076142132,
+    "step": 2280
+  },
+  {
+    "loss": 0.0336,
+    "grad_norm": 0.2884531319141388,
+    "learning_rate": 4.232016401228458e-05,
+    "epoch": 1.4530456852791878,
+    "step": 2290
+  },
+  {
+    "loss": 0.0346,
+    "grad_norm": 0.3362226188182831,
+    "learning_rate": 4.1418811469594884e-05,
+    "epoch": 1.4593908629441623,
+    "step": 2300
+  },
+  {
+    "loss": 0.0326,
+    "grad_norm": 0.3298276364803314,
+    "learning_rate": 4.05246474724709e-05,
+    "epoch": 1.465736040609137,
+    "step": 2310
+  },
+  {
+    "loss": 0.0412,
+    "grad_norm": 0.29744839668273926,
+    "learning_rate": 3.9637781744508205e-05,
+    "epoch": 1.4720812182741116,
+    "step": 2320
+  },
+  {
+    "loss": 0.0313,
+    "grad_norm": 0.16129061579704285,
+    "learning_rate": 3.875832311372568e-05,
+    "epoch": 1.4784263959390862,
+    "step": 2330
+  },
+  {
+    "loss": 0.0392,
+    "grad_norm": 0.4510703980922699,
+    "learning_rate": 3.788637949921141e-05,
+    "epoch": 1.484771573604061,
+    "step": 2340
+  },
+  {
+    "loss": 0.032,
+    "grad_norm": 0.29818281531333923,
+    "learning_rate": 3.70220578978795e-05,
+    "epoch": 1.4911167512690355,
+    "step": 2350
+  },
+  {
+    "loss": 0.0427,
+    "grad_norm": 0.48012402653694153,
+    "learning_rate": 3.616546437134083e-05,
+    "epoch": 1.49746192893401,
+    "step": 2360
+  },
+  {
+    "loss": 0.0311,
+    "grad_norm": 0.29981890320777893,
+    "learning_rate": 3.531670403288766e-05,
+    "epoch": 1.5038071065989849,
+    "step": 2370
+  },
+  {
+    "loss": 0.0346,
+    "grad_norm": 0.5202313661575317,
+    "learning_rate": 3.447588103459549e-05,
+    "epoch": 1.5101522842639594,
+    "step": 2380
+  },
+  {
+    "loss": 0.0395,
+    "grad_norm": 0.33824649453163147,
+    "learning_rate": 3.3643098554542195e-05,
+    "epoch": 1.516497461928934,
+    "step": 2390
+  },
+  {
+    "loss": 0.0337,
+    "grad_norm": 0.40760862827301025,
+    "learning_rate": 3.281845878414719e-05,
+    "epoch": 1.5228426395939088,
+    "step": 2400
+  },
+  {
+    "loss": 0.0411,
+    "grad_norm": 0.41836321353912354,
+    "learning_rate": 3.2002062915631306e-05,
+    "epoch": 1.529187817258883,
+    "step": 2410
+  },
+  {
+    "loss": 0.0353,
+    "grad_norm": 0.41234228014945984,
+    "learning_rate": 3.1194011129599324e-05,
+    "epoch": 1.5355329949238579,
+    "step": 2420
+  },
+  {
+    "loss": 0.0358,
+    "grad_norm": 0.4182668924331665,
+    "learning_rate": 3.0394402582746996e-05,
+    "epoch": 1.5418781725888326,
+    "step": 2430
+  },
+  {
+    "loss": 0.0293,
+    "grad_norm": 0.24028272926807404,
+    "learning_rate": 2.9603335395693143e-05,
+    "epoch": 1.548223350253807,
+    "step": 2440
+  },
+  {
+    "loss": 0.0373,
+    "grad_norm": 0.4068399667739868,
+    "learning_rate": 2.882090664093945e-05,
+    "epoch": 1.5545685279187818,
+    "step": 2450
+  },
+  {
+    "loss": 0.0343,
+    "grad_norm": 0.4273289442062378,
+    "learning_rate": 2.8047212330958474e-05,
+    "epoch": 1.5609137055837563,
+    "step": 2460
+  },
+  {
+    "loss": 0.0351,
+    "grad_norm": 0.346050888299942,
+    "learning_rate": 2.7282347406411836e-05,
+    "epoch": 1.5672588832487309,
+    "step": 2470
+  },
+  {
+    "loss": 0.0344,
+    "grad_norm": 0.19225843250751495,
+    "learning_rate": 2.6526405724500193e-05,
+    "epoch": 1.5736040609137056,
+    "step": 2480
+  },
+  {
+    "loss": 0.0291,
+    "grad_norm": 0.23613357543945312,
+    "learning_rate": 2.5779480047445627e-05,
+    "epoch": 1.5799492385786802,
+    "step": 2490
+  },
+  {
+    "loss": 0.0444,
+    "grad_norm": 0.26933079957962036,
+    "learning_rate": 2.5041662031109047e-05,
+    "epoch": 1.5862944162436547,
+    "step": 2500
+  },
+  {
+    "loss": 0.0363,
+    "grad_norm": 0.3984769582748413,
+    "learning_rate": 2.4313042213742744e-05,
+    "epoch": 1.5926395939086295,
+    "step": 2510
+  },
+  {
+    "loss": 0.035,
+    "grad_norm": 0.24797454476356506,
+    "learning_rate": 2.3593710004880565e-05,
+    "epoch": 1.598984771573604,
+    "step": 2520
+  },
+  {
+    "eval_loss": 0.05244370922446251,
+    "eval_runtime": 23.8874,
+    "eval_samples_per_second": 55.552,
+    "eval_steps_per_second": 13.899,
+    "epoch": 1.598984771573604,
+    "step": 2520
+  },
+  {
+    "loss": 0.0306,
+    "grad_norm": 0.2291479855775833,
+    "learning_rate": 2.2883753674366315e-05,
+    "epoch": 1.6053299492385786,
+    "step": 2530
+  },
+  {
+    "loss": 0.037,
+    "grad_norm": 0.3272293210029602,
+    "learning_rate": 2.2183260341522026e-05,
+    "epoch": 1.6116751269035534,
+    "step": 2540
+  },
+  {
+    "loss": 0.0297,
+    "grad_norm": 0.18494157493114471,
+    "learning_rate": 2.149231596445762e-05,
+    "epoch": 1.618020304568528,
+    "step": 2550
+  },
+  {
+    "loss": 0.0342,
+    "grad_norm": 0.25217947363853455,
+    "learning_rate": 2.0811005329522825e-05,
+    "epoch": 1.6243654822335025,
+    "step": 2560
+  },
+  {
+    "loss": 0.0407,
+    "grad_norm": 0.305937796831131,
+    "learning_rate": 2.0139412040903016e-05,
+    "epoch": 1.6307106598984773,
+    "step": 2570
+  },
+  {
+    "loss": 0.0346,
+    "grad_norm": 0.3220507502555847,
+    "learning_rate": 1.9477618510359964e-05,
+    "epoch": 1.6370558375634516,
+    "step": 2580
+  },
+  {
+    "loss": 0.0349,
+    "grad_norm": 0.2863691747188568,
+    "learning_rate": 1.882570594711919e-05,
+    "epoch": 1.6434010152284264,
+    "step": 2590
+  },
+  {
+    "loss": 0.0333,
+    "grad_norm": 0.7027552127838135,
+    "learning_rate": 1.8183754347904646e-05,
+    "epoch": 1.649746192893401,
+    "step": 2600
+  },
+  {
+    "loss": 0.0354,
+    "grad_norm": 0.43354132771492004,
+    "learning_rate": 1.755184248712215e-05,
+    "epoch": 1.6560913705583755,
+    "step": 2610
+  },
+  {
+    "loss": 0.0346,
+    "grad_norm": 0.3740198612213135,
+    "learning_rate": 1.6930047907193114e-05,
+    "epoch": 1.6624365482233503,
+    "step": 2620
+  },
+  {
+    "loss": 0.0412,
+    "grad_norm": 0.40340694785118103,
+    "learning_rate": 1.631844690903904e-05,
+    "epoch": 1.6687817258883249,
+    "step": 2630
+  },
+  {
+    "loss": 0.0309,
+    "grad_norm": 0.3262211084365845,
+    "learning_rate": 1.571711454271877e-05,
+    "epoch": 1.6751269035532994,
+    "step": 2640
+  },
+  {
+    "loss": 0.0333,
+    "grad_norm": 0.49483418464660645,
+    "learning_rate": 1.5126124598218826e-05,
+    "epoch": 1.6814720812182742,
+    "step": 2650
+  },
+  {
+    "loss": 0.0347,
+    "grad_norm": 0.34749022126197815,
+    "learning_rate": 1.4545549596398789e-05,
+    "epoch": 1.6878172588832487,
+    "step": 2660
+  },
+  {
+    "loss": 0.0256,
+    "grad_norm": 0.2993925213813782,
+    "learning_rate": 1.3975460780092053e-05,
+    "epoch": 1.6941624365482233,
+    "step": 2670
+  },
+  {
+    "loss": 0.0432,
+    "grad_norm": 0.21877335011959076,
+    "learning_rate": 1.3415928105363595e-05,
+    "epoch": 1.700507614213198,
+    "step": 2680
+  },
+  {
+    "loss": 0.0363,
+    "grad_norm": 0.4628012180328369,
+    "learning_rate": 1.2867020232925675e-05,
+    "epoch": 1.7068527918781726,
+    "step": 2690
+  },
+  {
+    "loss": 0.0283,
+    "grad_norm": 0.34107235074043274,
+    "learning_rate": 1.2328804519712312e-05,
+    "epoch": 1.7131979695431472,
+    "step": 2700
+  },
+  {
+    "loss": 0.0358,
+    "grad_norm": 0.38295891880989075,
+    "learning_rate": 1.1801347010613962e-05,
+    "epoch": 1.719543147208122,
+    "step": 2710
+  },
+  {
+    "loss": 0.0303,
+    "grad_norm": 0.3934635519981384,
+    "learning_rate": 1.1284712430373056e-05,
+    "epoch": 1.7258883248730963,
+    "step": 2720
+  },
+  {
+    "loss": 0.0355,
+    "grad_norm": 0.4926978051662445,
+    "learning_rate": 1.0778964175641548e-05,
+    "epoch": 1.732233502538071,
+    "step": 2730
+  },
+  {
+    "loss": 0.0289,
+    "grad_norm": 0.2570490837097168,
+    "learning_rate": 1.0284164307201493e-05,
+    "epoch": 1.7385786802030458,
+    "step": 2740
+  },
+  {
+    "loss": 0.0366,
+    "grad_norm": 0.4256710708141327,
+    "learning_rate": 9.800373542349483e-06,
+    "epoch": 1.7449238578680202,
+    "step": 2750
+  },
+  {
+    "loss": 0.0303,
+    "grad_norm": 0.31267115473747253,
+    "learning_rate": 9.32765124744608e-06,
+    "epoch": 1.751269035532995,
+    "step": 2760
+  },
+  {
+    "loss": 0.0406,
+    "grad_norm": 0.2698454260826111,
+    "learning_rate": 8.86605543063076e-06,
+    "epoch": 1.7576142131979695,
+    "step": 2770
+  },
+  {
+    "loss": 0.0348,
+    "grad_norm": 0.2938295006752014,
+    "learning_rate": 8.415642734703821e-06,
+    "epoch": 1.763959390862944,
+    "step": 2780
+  },
+  {
+    "loss": 0.03,
+    "grad_norm": 0.3625064790248871,
+    "learning_rate": 7.976468430175599e-06,
+    "epoch": 1.7703045685279188,
+    "step": 2790
+  },
+  {
+    "loss": 0.0264,
+    "grad_norm": 0.35078808665275574,
+    "learning_rate": 7.548586408484282e-06,
+    "epoch": 1.7766497461928934,
+    "step": 2800
+  },
+  {
+    "loss": 0.0287,
+    "grad_norm": 0.30412939190864563,
+    "learning_rate": 7.1320491753827825e-06,
+    "epoch": 1.782994923857868,
+    "step": 2810
+  },
+  {
+    "loss": 0.0299,
+    "grad_norm": 0.33658623695373535,
+    "learning_rate": 6.726907844495689e-06,
+    "epoch": 1.7893401015228427,
+    "step": 2820
+  },
+  {
+    "loss": 0.0336,
+    "grad_norm": 0.34409451484680176,
+    "learning_rate": 6.333212131047162e-06,
+    "epoch": 1.7956852791878173,
+    "step": 2830
+  },
+  {
+    "eval_loss": 0.05158456787467003,
+    "eval_runtime": 23.7884,
+    "eval_samples_per_second": 55.784,
+    "eval_steps_per_second": 13.956,
+    "epoch": 1.7988578680203045,
+    "step": 2835
+  },
+  {
+    "loss": 0.0359,
+    "grad_norm": 0.45101943612098694,
+    "learning_rate": 5.951010345760221e-06,
+    "epoch": 1.8020304568527918,
+    "step": 2840
+  },
+  {
+    "loss": 0.0335,
+    "grad_norm": 0.2589711844921112,
+    "learning_rate": 5.580349388928607e-06,
+    "epoch": 1.8083756345177666,
+    "step": 2850
+  },
+  {
+    "loss": 0.038,
+    "grad_norm": 0.2490888088941574,
+    "learning_rate": 5.221274744661553e-06,
+    "epoch": 1.8147208121827412,
+    "step": 2860
+  },
+  {
+    "loss": 0.0305,
+    "grad_norm": 0.3543383777141571,
+    "learning_rate": 4.873830475302377e-06,
+    "epoch": 1.8210659898477157,
+    "step": 2870
+  },
+  {
+    "loss": 0.0327,
+    "grad_norm": 0.26961830258369446,
+    "learning_rate": 4.538059216021595e-06,
+    "epoch": 1.8274111675126905,
+    "step": 2880
+  },
+  {
+    "loss": 0.031,
+    "grad_norm": 0.34061428904533386,
+    "learning_rate": 4.214002169585107e-06,
+    "epoch": 1.8337563451776648,
+    "step": 2890
+  },
+  {
+    "loss": 0.0255,
+    "grad_norm": 0.30612456798553467,
+    "learning_rate": 3.901699101298173e-06,
+    "epoch": 1.8401015228426396,
+    "step": 2900
+  },
+  {
+    "loss": 0.0293,
+    "grad_norm": 0.34559229016304016,
+    "learning_rate": 3.60118833412576e-06,
+    "epoch": 1.8464467005076142,
+    "step": 2910
+  },
+  {
+    "loss": 0.0303,
+    "grad_norm": 0.3501076400279999,
+    "learning_rate": 3.3125067439899406e-06,
+    "epoch": 1.8527918781725887,
+    "step": 2920
+  },
+  {
+    "loss": 0.0262,
+    "grad_norm": 0.560280442237854,
+    "learning_rate": 3.0356897552448194e-06,
+    "epoch": 1.8591370558375635,
+    "step": 2930
+  },
+  {
+    "loss": 0.029,
+    "grad_norm": 0.23615562915802002,
+    "learning_rate": 2.7707713363294806e-06,
+    "epoch": 1.865482233502538,
+    "step": 2940
+  },
+  {
+    "loss": 0.0303,
+    "grad_norm": 0.22095191478729248,
+    "learning_rate": 2.517783995599865e-06,
+    "epoch": 1.8718274111675126,
+    "step": 2950
+  },
+  {
+    "loss": 0.0337,
+    "grad_norm": 0.5112082958221436,
+    "learning_rate": 2.2767587773394605e-06,
+    "epoch": 1.8781725888324874,
+    "step": 2960
+  },
+  {
+    "loss": 0.0351,
+    "grad_norm": 0.3294392228126526,
+    "learning_rate": 2.0477252579500062e-06,
+    "epoch": 1.884517766497462,
+    "step": 2970
+  },
+  {
+    "loss": 0.0348,
+    "grad_norm": 0.5974733233451843,
+    "learning_rate": 1.8307115423219945e-06,
+    "epoch": 1.8908629441624365,
+    "step": 2980
+  },
+  {
+    "loss": 0.0303,
+    "grad_norm": 0.3671284019947052,
+    "learning_rate": 1.6257442603860196e-06,
+    "epoch": 1.8972081218274113,
+    "step": 2990
+  },
+  {
+    "loss": 0.0341,
+    "grad_norm": 0.3103559911251068,
+    "learning_rate": 1.4328485638449018e-06,
+    "epoch": 1.9035532994923858,
+    "step": 3000
+  },
+  {
+    "loss": 0.0322,
+    "grad_norm": 0.22946634888648987,
+    "learning_rate": 1.252048123087346e-06,
+    "epoch": 1.9098984771573604,
+    "step": 3010
+  },
+  {
+    "loss": 0.0299,
+    "grad_norm": 0.4102902114391327,
+    "learning_rate": 1.0833651242833643e-06,
+    "epoch": 1.9162436548223352,
+    "step": 3020
+  },
+  {
+    "loss": 0.0359,
+    "grad_norm": 0.49926474690437317,
+    "learning_rate": 9.268202666617099e-07,
+    "epoch": 1.9225888324873095,
+    "step": 3030
+  },
+  {
+    "loss": 0.0368,
+    "grad_norm": 0.48568469285964966,
+    "learning_rate": 7.824327599699199e-07,
+    "epoch": 1.9289340101522843,
+    "step": 3040
+  },
+  {
+    "loss": 0.0269,
+    "grad_norm": 0.4492437541484833,
+    "learning_rate": 6.502203221170677e-07,
+    "epoch": 1.9352791878172588,
+    "step": 3050
+  },
+  {
+    "loss": 0.0396,
+    "grad_norm": 0.25819239020347595,
+    "learning_rate": 5.301991769995352e-07,
+    "epoch": 1.9416243654822334,
+    "step": 3060
+  },
+  {
+    "loss": 0.0301,
+    "grad_norm": 0.24553845822811127,
+    "learning_rate": 4.223840525102385e-07,
+    "epoch": 1.9479695431472082,
+    "step": 3070
+  },
+  {
+    "loss": 0.0271,
+    "grad_norm": 0.4226374924182892,
+    "learning_rate": 3.267881787312854e-07,
+    "epoch": 1.9543147208121827,
+    "step": 3080
+  },
+  {
+    "loss": 0.0334,
+    "grad_norm": 0.5136414766311646,
+    "learning_rate": 2.434232863105623e-07,
+    "epoch": 1.9606598984771573,
+    "step": 3090
+  },
+  {
+    "loss": 0.0258,
+    "grad_norm": 0.3708105981349945,
+    "learning_rate": 1.722996050222081e-07,
+    "epoch": 1.967005076142132,
+    "step": 3100
+  },
+  {
+    "loss": 0.0363,
+    "grad_norm": 0.28896448016166687,
+    "learning_rate": 1.1342586251132936e-07,
+    "epoch": 1.9733502538071066,
+    "step": 3110
+  },
+  {
+    "loss": 0.032,
+    "grad_norm": 0.2560702860355377,
+    "learning_rate": 6.680928322302382e-08,
+    "epoch": 1.9796954314720812,
+    "step": 3120
+  },
+  {
+    "loss": 0.0312,
+    "grad_norm": 0.5401180386543274,
+    "learning_rate": 3.2455587515889394e-08,
+    "epoch": 1.986040609137056,
+    "step": 3130
+  },
+  {
+    "loss": 0.0319,
+    "grad_norm": 0.5009672045707703,
+    "learning_rate": 1.0368990960019088e-08,
+    "epoch": 1.9923857868020305,
+    "step": 3140
+  },
+  {
+    "loss": 0.0253,
+    "grad_norm": 0.2637391984462738,
+    "learning_rate": 5.522038197369739e-10,
+    "epoch": 1.998730964467005,
+    "step": 3150
+  },
+  {
+    "eval_loss": 0.051274195313453674,
+    "eval_runtime": 23.5282,
+    "eval_samples_per_second": 56.4,
+    "eval_steps_per_second": 14.111,
+    "epoch": 1.998730964467005,
+    "step": 3150
+  },
+  {
+    "train_runtime": 1163.1262,
+    "train_samples_per_second": 43.335,
+    "train_steps_per_second": 2.71,
+    "total_flos": 8.249162336833536e+16,
+    "train_loss": 0.0777996052304746,
+    "epoch": 2.0,
+    "step": 3152
+  }
+]