Upload folder using huggingface_hub

README.md

@@ -0,0 +1,88 @@
+---
+license: other
+license_name: lfm1.0
+license_link: https://huggingface.co/LiquidAI/LFM2.5-350M/blob/main/LICENSE
+base_model: LiquidAI/LFM2.5-350M
+tags:
+  - lfm2
+  - lfm2.5
+  - liquid
+  - python
+  - math
+  - fine-tune
+language:
+  - en
+pipeline_tag: text-generation
+---
+
+# LFM2.5-350M-Python-Math
+
+A fine-tune of [LiquidAI/LFM2.5-350M](https://huggingface.co/LiquidAI/LFM2.5-350M) (instruct) focused on **Python code generation** and **math word-problem solving**, while retaining general chat ability through a balanced mixed dataset.
+
+## Why this exists
+
+The previous 230M fine-tune (`lfm2.5-230m-code-math`) showed strong potential but suffered from catastrophic forgetting (e.g., confusing baking cookies with HTTP cookies, failing negative constraints like "no dairy"). This 350M version addresses those issues by:
+
+1. **Mixing general chat data** (`yahma/alpaca-cleaned`, 30k samples) to prevent knowledge loss.
+2. **Injecting custom fix-it examples** targeting specific failure modes (negative constraints, complete Pygame scripts).
+3. **Using longer context** (2048 tokens) so code outputs aren't truncated mid-function.
+4. **Reducing epochs to 2** with a lower learning rate (`2e-5`) to prevent overfitting observed in earlier runs.
+
+Fine-tuning started from the **instruct checkpoint** rather than base. Testing confirmed that at 350M scale, starting from base with a mixed dataset still produced alignment failures (refusals, identity confusion, math regression), while the instruct checkpoint with the same dataset produced consistently strong results.
+
+## Training details
+
+- **Base model:** `LiquidAI/LFM2.5-350M` (instruct)
+- **Method:** Full fine-tune (96GB VRAM, no LoRA needed)
+- **Datasets:**
+  - Python Code: `iamtarun/python_code_instructions_18k_alpaca` (Python-focused, replacing the multi-language 120k set)
+  - Math: `openai/gsm8k` (main split)
+  - General Chat: `yahma/alpaca-cleaned` (30k sample subset)
+  - Custom Fix-It: Hand-crafted examples for negative constraints ("no dairy", "no eggs") and complete runnable Pygame scripts (duplicated 50× for weight)
+- **Checkpoint selection:** Best by eval_loss
+- **Sequence length:** 2048 tokens (increased from 1024 to accommodate full scripts)
+- **Max response chars:** 3500 (prevents code truncation)
+- **Epochs:** 2 (reduced from 4; overfitting observed past epoch 2 in prior runs)
+- **Learning rate:** 2e-5 (reduced from 5e-5 for 350M stability)
+- **Loss:** Completion-only
+
+## What it's good at
+
+- **Python Code:** Complete, runnable scripts including Pygame game loops, file I/O, classes, list comprehensions, and algorithmic implementations (e.g., two-pointer palindrome check). No more placeholder `pass` statements or truncated functions.
+- **Math:** GSM8K-style word problems with step-by-step reasoning annotations (`<<...>>`). Reliable on algebra, percentages, geometry, and multi-step arithmetic.
+- **General Chat:** Retains coherent conversational ability. Correctly handles negative constraints (e.g., "breakfast without eggs" returns egg-free options). Knows the difference between baking cookies and browser cookies.
+- **Speed:** At 350M parameters, achieves ~157 t/s generation on laptop CPU (i5-12450H) with Q5_K_S quantization via llama.cpp.
+
+## Known limitations
+
+- **Python only:** Trained exclusively on Python code instructions. Other languages were not included in this fine-tune.
+- **Sentence counting:** May not strictly adhere to "exactly N sentences" constraints.
+- **Identity:** May occasionally claim to be developed by Google (artifact from Alpaca-Cleaned training data).
+- **Still 350M parameters:** Do not expect deep multi-step reasoning or long-form creative writing at the level of larger models.
+- Not evaluated on safety-critical, medical, or legal use cases.
+
+## Usage
+
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+model_id = "hauser458original/lfm2.5-350m-python-math"
+model = AutoModelForCausalLM.from_pretrained(model_id, dtype="bfloat16", device_map="auto")
+tokenizer = AutoTokenizer.from_pretrained(model_id)
+
+messages = [{"role": "user", "content": "Write a Python function to check if a number is prime."}]
+inputs = tokenizer.apply_chat_template(
+    messages, add_generation_prompt=True, return_tensors="pt", return_dict=True
+).to(model.device)
+
+output = model.generate(**inputs, max_new_tokens=512, do_sample=True, temperature=0.5, top_p=0.9)
+print(tokenizer.decode(output[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True))
+
+GGUF quantized versions (Q4_K_M, Q5_K_S, Q5_K_M, Q8_0, F16) for llama.cpp/Ollama/LM Studio are available at: hauser458original/lfm2.5-350m-python-math-GGUF
+
+## License
+
+Inherits the [LFM Open License v1.0](https://huggingface.co/LiquidAI/LFM2.5-350M/blob/main/LICENSE) from the base model.
+
+## Acknowledgements
+
+Built on [LiquidAI/LFM2.5-350M](https://huggingface.co/LiquidAI/LFM2.5-350M). See the [LFM2 Technical Report](https://arxiv.org/abs/2511.23404) for details on the base architecture.

chat_template.jinja

@@ -0,0 +1,64 @@
+{{- bos_token -}}
+{%- set keep_past_thinking = keep_past_thinking | default(false) -%}
+{%- set ns = namespace(system_prompt="") -%}
+{%- if messages[0]["role"] == "system" -%}
+    {%- set sys_content = messages[0]["content"] -%}
+    {%- if sys_content is not string -%}
+        {%- for item in sys_content -%}
+            {%- if item["type"] == "text" -%}
+                {%- set ns.system_prompt = ns.system_prompt + item["text"] -%}
+            {%- endif -%}
+        {%- endfor -%}
+    {%- else -%}
+        {%- set ns.system_prompt = sys_content -%}
+    {%- endif -%}
+    {%- set messages = messages[1:] -%}
+{%- endif -%}
+{%- if tools -%}
+    {%- set ns.system_prompt = ns.system_prompt + ("\n" if ns.system_prompt else "") + "List of tools: [" -%}
+    {%- for tool in tools -%}
+        {%- if tool is not string -%}
+            {%- set tool = tool | tojson -%}
+        {%- endif -%}
+        {%- set ns.system_prompt = ns.system_prompt + tool -%}
+        {%- if not loop.last -%}
+            {%- set ns.system_prompt = ns.system_prompt + ", " -%}
+        {%- endif -%}
+    {%- endfor -%}
+    {%- set ns.system_prompt = ns.system_prompt + "]" -%}
+{%- endif -%}
+{%- if ns.system_prompt -%}
+    {{- "<|im_start|>system\n" + ns.system_prompt + "<|im_end|>\n" -}}
+{%- endif -%}
+{%- set ns.last_assistant_index = -1 -%}
+{%- for message in messages -%}
+    {%- if message["role"] == "assistant" -%}
+        {%- set ns.last_assistant_index = loop.index0 -%}
+    {%- endif -%}
+{%- endfor -%}
+{%- for message in messages -%}
+    {{- "<|im_start|>" + message["role"] + "\n" -}}
+    {%- set content = message["content"] -%}
+    {%- if content is not string -%}
+        {%- set ns.content = "" -%}
+        {%- for item in content -%}
+            {%- if item["type"] == "image" -%}
+                {%- set ns.content = ns.content + "<image>" -%}
+            {%- elif item["type"] == "text" -%}
+                {%- set ns.content = ns.content + item["text"] -%}
+            {%- else -%}
+                {%- set ns.content = ns.content + item | tojson -%}
+            {%- endif -%}
+        {%- endfor -%}
+        {%- set content = ns.content -%}
+    {%- endif -%}
+    {%- if message["role"] == "assistant" and not keep_past_thinking and loop.index0 != ns.last_assistant_index -%}
+        {%- if "</think>" in content -%}
+            {%- set content = content.split("</think>")[-1] | trim -%}
+        {%- endif -%}
+    {%- endif -%}
+    {{- content + "<|im_end|>\n" -}}
+{%- endfor -%}
+{%- if add_generation_prompt -%}
+    {{- "<|im_start|>assistant\n" -}}
+{%- endif -%}

checkpoint-2000/chat_template.jinja

@@ -0,0 +1,64 @@
+{{- bos_token -}}
+{%- set keep_past_thinking = keep_past_thinking | default(false) -%}
+{%- set ns = namespace(system_prompt="") -%}
+{%- if messages[0]["role"] == "system" -%}
+    {%- set sys_content = messages[0]["content"] -%}
+    {%- if sys_content is not string -%}
+        {%- for item in sys_content -%}
+            {%- if item["type"] == "text" -%}
+                {%- set ns.system_prompt = ns.system_prompt + item["text"] -%}
+            {%- endif -%}
+        {%- endfor -%}
+    {%- else -%}
+        {%- set ns.system_prompt = sys_content -%}
+    {%- endif -%}
+    {%- set messages = messages[1:] -%}
+{%- endif -%}
+{%- if tools -%}
+    {%- set ns.system_prompt = ns.system_prompt + ("\n" if ns.system_prompt else "") + "List of tools: [" -%}
+    {%- for tool in tools -%}
+        {%- if tool is not string -%}
+            {%- set tool = tool | tojson -%}
+        {%- endif -%}
+        {%- set ns.system_prompt = ns.system_prompt + tool -%}
+        {%- if not loop.last -%}
+            {%- set ns.system_prompt = ns.system_prompt + ", " -%}
+        {%- endif -%}
+    {%- endfor -%}
+    {%- set ns.system_prompt = ns.system_prompt + "]" -%}
+{%- endif -%}
+{%- if ns.system_prompt -%}
+    {{- "<|im_start|>system\n" + ns.system_prompt + "<|im_end|>\n" -}}
+{%- endif -%}
+{%- set ns.last_assistant_index = -1 -%}
+{%- for message in messages -%}
+    {%- if message["role"] == "assistant" -%}
+        {%- set ns.last_assistant_index = loop.index0 -%}
+    {%- endif -%}
+{%- endfor -%}
+{%- for message in messages -%}
+    {{- "<|im_start|>" + message["role"] + "\n" -}}
+    {%- set content = message["content"] -%}
+    {%- if content is not string -%}
+        {%- set ns.content = "" -%}
+        {%- for item in content -%}
+            {%- if item["type"] == "image" -%}
+                {%- set ns.content = ns.content + "<image>" -%}
+            {%- elif item["type"] == "text" -%}
+                {%- set ns.content = ns.content + item["text"] -%}
+            {%- else -%}
+                {%- set ns.content = ns.content + item | tojson -%}
+            {%- endif -%}
+        {%- endfor -%}
+        {%- set content = ns.content -%}
+    {%- endif -%}
+    {%- if message["role"] == "assistant" and not keep_past_thinking and loop.index0 != ns.last_assistant_index -%}
+        {%- if "</think>" in content -%}
+            {%- set content = content.split("</think>")[-1] | trim -%}
+        {%- endif -%}
+    {%- endif -%}
+    {{- content + "<|im_end|>\n" -}}
+{%- endfor -%}
+{%- if add_generation_prompt -%}
+    {{- "<|im_start|>assistant\n" -}}
+{%- endif -%}

checkpoint-2000/config.json

@@ -0,0 +1,60 @@
+{
+  "architectures": [
+    "Lfm2ForCausalLM"
+  ],
+  "block_auto_adjust_ff_dim": true,
+  "block_dim": 1024,
+  "block_ffn_dim_multiplier": 1.0,
+  "block_mlp_init_scale": 1.0,
+  "block_multiple_of": 256,
+  "block_norm_eps": 1e-05,
+  "block_out_init_scale": 1.0,
+  "block_use_swiglu": true,
+  "block_use_xavier_init": true,
+  "bos_token_id": 1,
+  "conv_L_cache": 3,
+  "conv_bias": false,
+  "conv_dim": 1024,
+  "conv_use_xavier_init": true,
+  "dtype": "bfloat16",
+  "eos_token_id": 7,
+  "full_attn_idxs": null,
+  "hidden_size": 1024,
+  "initializer_range": 0.02,
+  "intermediate_size": 6656,
+  "layer_types": [
+    "conv",
+    "conv",
+    "full_attention",
+    "conv",
+    "conv",
+    "full_attention",
+    "conv",
+    "conv",
+    "full_attention",
+    "conv",
+    "full_attention",
+    "conv",
+    "full_attention",
+    "conv",
+    "full_attention",
+    "conv"
+  ],
+  "max_position_embeddings": 128000,
+  "model_type": "lfm2",
+  "norm_eps": 1e-05,
+  "num_attention_heads": 16,
+  "num_heads": 16,
+  "num_hidden_layers": 16,
+  "num_key_value_heads": 8,
+  "pad_token_id": 0,
+  "rope_parameters": {
+    "rope_theta": 1000000.0,
+    "rope_type": "default"
+  },
+  "tie_word_embeddings": true,
+  "transformers_version": "5.5.0",
+  "use_cache": false,
+  "use_pos_enc": true,
+  "vocab_size": 65536
+}

checkpoint-2000/generation_config.json

@@ -0,0 +1,9 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 1,
+  "eos_token_id": [
+    7
+  ],
+  "pad_token_id": 0,
+  "transformers_version": "5.5.0"
+}

checkpoint-2000/model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9393f975e13b7b4f0ddbb9081a5ab9986b609333524c504e9dcd68109beeaa02
+size 708984464

checkpoint-2000/optimizer.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:abab03a522ed7a9dcd14e897285e1b6f55f2188a706edab4c6139e1822a86f40
+size 1418061003

checkpoint-2000/rng_state.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d1f3e429ffab9361eb588f03657ab12e499db270bb2234e5408b66a7fc8b7a88
+size 14645

checkpoint-2000/scheduler.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ab7ca6654dfd806f099bda93354c2523f7dd63e9b881b37f549eb66d0ea17011
+size 1465

checkpoint-2000/tokenizer_config.json

@@ -0,0 +1,21 @@
+{
+  "backend": "tokenizers",
+  "bos_token": "<|startoftext|>",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|im_end|>",
+  "extra_special_tokens": [],
+  "is_local": false,
+  "legacy": false,
+  "model_input_names": [
+    "input_ids",
+    "attention_mask"
+  ],
+  "model_max_length": 1000000000000000019884624838656,
+  "model_specific_special_tokens": {},
+  "pad_token": "<|pad|>",
+  "sp_model_kwargs": {},
+  "spaces_between_special_tokens": false,
+  "tokenizer_class": "TokenizersBackend",
+  "use_default_system_prompt": false,
+  "use_fast": true
+}

checkpoint-2000/trainer_state.json

@@ -0,0 +1,944 @@
+{
+  "best_global_step": 2000,
+  "best_metric": 0.8990073204040527,
+  "best_model_checkpoint": "lfm2.5-350m-balanced/checkpoint-2000",
+  "epoch": 1.4652014652014653,
+  "eval_steps": 200,
+  "global_step": 2000,
+  "is_hyper_param_search": false,
+  "is_local_process_zero": true,
+  "is_world_process_zero": true,
+  "log_history": [
+    {
+      "entropy": 1.20119291305542,
+      "epoch": 0.018315018315018316,
+      "grad_norm": 10.4375,
+      "learning_rate": 5.853658536585366e-06,
+      "loss": 2.8065673828125,
+      "mean_token_accuracy": 0.57585333943367,
+      "num_tokens": 148430.0,
+      "step": 25
+    },
+    {
+      "entropy": 1.278236083984375,
+      "epoch": 0.03663003663003663,
+      "grad_norm": 4.125,
+      "learning_rate": 1.1951219512195123e-05,
+      "loss": 1.506644287109375,
+      "mean_token_accuracy": 0.7009313237667084,
+      "num_tokens": 301448.0,
+      "step": 50
+    },
+    {
+      "entropy": 1.0903868675231934,
+      "epoch": 0.054945054945054944,
+      "grad_norm": 2.609375,
+      "learning_rate": 1.804878048780488e-05,
+      "loss": 1.0758729553222657,
+      "mean_token_accuracy": 0.7404501688480377,
+      "num_tokens": 458432.0,
+      "step": 75
+    },
+    {
+      "entropy": 1.0081040942668915,
+      "epoch": 0.07326007326007326,
+      "grad_norm": 2.4375,
+      "learning_rate": 1.9997966159416884e-05,
+      "loss": 0.9917147827148437,
+      "mean_token_accuracy": 0.7522138917446136,
+      "num_tokens": 616048.0,
+      "step": 100
+    },
+    {
+      "entropy": 0.9854830229282379,
+      "epoch": 0.09157509157509157,
+      "grad_norm": 2.46875,
+      "learning_rate": 1.99875879788391e-05,
+      "loss": 0.9723490905761719,
+      "mean_token_accuracy": 0.7565386164188385,
+      "num_tokens": 772466.0,
+      "step": 125
+    },
+    {
+      "entropy": 0.9745642852783203,
+      "epoch": 0.10989010989010989,
+      "grad_norm": 2.25,
+      "learning_rate": 1.996842417490666e-05,
+      "loss": 0.9625119018554688,
+      "mean_token_accuracy": 0.760125788450241,
+      "num_tokens": 930723.0,
+      "step": 150
+    },
+    {
+      "entropy": 0.9700245261192322,
+      "epoch": 0.1282051282051282,
+      "grad_norm": 2.234375,
+      "learning_rate": 1.99404916051395e-05,
+      "loss": 0.9550397491455078,
+      "mean_token_accuracy": 0.7578657758235932,
+      "num_tokens": 1091160.0,
+      "step": 175
+    },
+    {
+      "entropy": 0.9669338822364807,
+      "epoch": 0.14652014652014653,
+      "grad_norm": 2.359375,
+      "learning_rate": 1.9903814840538914e-05,
+      "loss": 0.9478360748291016,
+      "mean_token_accuracy": 0.7623070132732391,
+      "num_tokens": 1245781.0,
+      "step": 200
+    },
+    {
+      "epoch": 0.14652014652014653,
+      "eval_entropy": 0.9581337045464251,
+      "eval_loss": 0.9430390000343323,
+      "eval_mean_token_accuracy": 0.7639545847972234,
+      "eval_num_tokens": 1245781.0,
+      "eval_runtime": 8.0902,
+      "eval_samples_per_second": 284.171,
+      "eval_steps_per_second": 17.799,
+      "step": 200
+    },
+    {
+      "entropy": 0.9653169775009155,
+      "epoch": 0.16483516483516483,
+      "grad_norm": 2.640625,
+      "learning_rate": 1.9858426143973623e-05,
+      "loss": 0.9534239959716797,
+      "mean_token_accuracy": 0.7631052339076996,
+      "num_tokens": 1395650.0,
+      "step": 225
+    },
+    {
+      "entropy": 0.962496577501297,
+      "epoch": 0.18315018315018314,
+      "grad_norm": 2.234375,
+      "learning_rate": 1.9804365441799555e-05,
+      "loss": 0.9441399383544922,
+      "mean_token_accuracy": 0.7629920542240143,
+      "num_tokens": 1553420.0,
+      "step": 250
+    },
+    {
+      "entropy": 0.9607322406768799,
+      "epoch": 0.20146520146520147,
+      "grad_norm": 2.46875,
+      "learning_rate": 1.9741680288738495e-05,
+      "loss": 0.939505615234375,
+      "mean_token_accuracy": 0.7633682239055634,
+      "num_tokens": 1708083.0,
+      "step": 275
+    },
+    {
+      "entropy": 0.9339856076240539,
+      "epoch": 0.21978021978021978,
+      "grad_norm": 2.203125,
+      "learning_rate": 1.967042582604635e-05,
+      "loss": 0.9153070831298828,
+      "mean_token_accuracy": 0.767605767250061,
+      "num_tokens": 1862545.0,
+      "step": 300
+    },
+    {
+      "entropy": 0.9721270000934601,
+      "epoch": 0.23809523809523808,
+      "grad_norm": 2.265625,
+      "learning_rate": 1.9590664733007947e-05,
+      "loss": 0.959111328125,
+      "mean_token_accuracy": 0.7571033167839051,
+      "num_tokens": 2017595.0,
+      "step": 325
+    },
+    {
+      "entropy": 0.961563161611557,
+      "epoch": 0.2564102564102564,
+      "grad_norm": 2.3125,
+      "learning_rate": 1.9502467171800902e-05,
+      "loss": 0.9458185577392578,
+      "mean_token_accuracy": 0.7628096437454224,
+      "num_tokens": 2166320.0,
+      "step": 350
+    },
+    {
+      "entropy": 0.9491504514217377,
+      "epoch": 0.27472527472527475,
+      "grad_norm": 2.34375,
+      "learning_rate": 1.940591072577719e-05,
+      "loss": 0.9269615173339844,
+      "mean_token_accuracy": 0.7635479891300201,
+      "num_tokens": 2325678.0,
+      "step": 375
+    },
+    {
+      "entropy": 0.9450718343257904,
+      "epoch": 0.29304029304029305,
+      "grad_norm": 2.28125,
+      "learning_rate": 1.930108033121663e-05,
+      "loss": 0.9313880920410156,
+      "mean_token_accuracy": 0.7636011290550232,
+      "num_tokens": 2481053.0,
+      "step": 400
+    },
+    {
+      "epoch": 0.29304029304029305,
+      "eval_entropy": 0.9506676958666908,
+      "eval_loss": 0.9265689849853516,
+      "eval_mean_token_accuracy": 0.7673345183332761,
+      "eval_num_tokens": 2481053.0,
+      "eval_runtime": 8.0719,
+      "eval_samples_per_second": 284.816,
+      "eval_steps_per_second": 17.84,
+      "step": 400
+    },
+    {
+      "entropy": 0.9616206979751587,
+      "epoch": 0.31135531135531136,
+      "grad_norm": 2.28125,
+      "learning_rate": 1.9188068202612317e-05,
+      "loss": 0.9455902862548828,
+      "mean_token_accuracy": 0.7630793690681458,
+      "num_tokens": 2633845.0,
+      "step": 425
+    },
+    {
+      "entropy": 0.9392594003677368,
+      "epoch": 0.32967032967032966,
+      "grad_norm": 2.3125,
+      "learning_rate": 1.906697375155377e-05,
+      "loss": 0.9156442260742188,
+      "mean_token_accuracy": 0.7659864926338196,
+      "num_tokens": 2790745.0,
+      "step": 450
+    },
+    {
+      "entropy": 0.9245667338371277,
+      "epoch": 0.34798534798534797,
+      "grad_norm": 2.203125,
+      "learning_rate": 1.8937903499279104e-05,
+      "loss": 0.903341064453125,
+      "mean_token_accuracy": 0.7713142216205597,
+      "num_tokens": 2943444.0,
+      "step": 475
+    },
+    {
+      "entropy": 0.912586669921875,
+      "epoch": 0.3663003663003663,
+      "grad_norm": 2.390625,
+      "learning_rate": 1.880097098297319e-05,
+      "loss": 0.8922891235351562,
+      "mean_token_accuracy": 0.771296215057373,
+      "num_tokens": 3098959.0,
+      "step": 500
+    },
+    {
+      "entropy": 0.9415633583068848,
+      "epoch": 0.38461538461538464,
+      "grad_norm": 2.15625,
+      "learning_rate": 1.8656296655894205e-05,
+      "loss": 0.9213317108154296,
+      "mean_token_accuracy": 0.7668669641017913,
+      "num_tokens": 3248382.0,
+      "step": 525
+    },
+    {
+      "entropy": 0.9617620420455932,
+      "epoch": 0.40293040293040294,
+      "grad_norm": 2.109375,
+      "learning_rate": 1.8504007781416385e-05,
+      "loss": 0.9434718322753907,
+      "mean_token_accuracy": 0.7617218482494355,
+      "num_tokens": 3409803.0,
+      "step": 550
+    },
+    {
+      "entropy": 0.9193656361103058,
+      "epoch": 0.42124542124542125,
+      "grad_norm": 2.0625,
+      "learning_rate": 1.8344238321082316e-05,
+      "loss": 0.901815185546875,
+      "mean_token_accuracy": 0.7707051420211792,
+      "num_tokens": 3559416.0,
+      "step": 575
+    },
+    {
+      "entropy": 0.9518638348579407,
+      "epoch": 0.43956043956043955,
+      "grad_norm": 2.15625,
+      "learning_rate": 1.817712881676307e-05,
+      "loss": 0.9327508544921875,
+      "mean_token_accuracy": 0.7644142961502075,
+      "num_tokens": 3710110.0,
+      "step": 600
+    },
+    {
+      "epoch": 0.43956043956043955,
+      "eval_entropy": 0.9517907314002514,
+      "eval_loss": 0.9172877073287964,
+      "eval_mean_token_accuracy": 0.7684686299827364,
+      "eval_num_tokens": 3710110.0,
+      "eval_runtime": 8.0887,
+      "eval_samples_per_second": 284.222,
+      "eval_steps_per_second": 17.803,
+      "step": 600
+    },
+    {
+      "entropy": 0.9130004584789276,
+      "epoch": 0.45787545787545786,
+      "grad_norm": 2.21875,
+      "learning_rate": 1.8002826267029977e-05,
+      "loss": 0.8941175079345703,
+      "mean_token_accuracy": 0.7705032932758331,
+      "num_tokens": 3862020.0,
+      "step": 625
+    },
+    {
+      "entropy": 0.9252553701400756,
+      "epoch": 0.47619047619047616,
+      "grad_norm": 2.15625,
+      "learning_rate": 1.7821483997846722e-05,
+      "loss": 0.9165068054199219,
+      "mean_token_accuracy": 0.7672811985015869,
+      "num_tokens": 4018231.0,
+      "step": 650
+    },
+    {
+      "entropy": 0.9462221479415893,
+      "epoch": 0.4945054945054945,
+      "grad_norm": 2.125,
+      "learning_rate": 1.763326152769551e-05,
+      "loss": 0.9259709167480469,
+      "mean_token_accuracy": 0.7658140981197357,
+      "num_tokens": 4173058.0,
+      "step": 675
+    },
+    {
+      "entropy": 0.9479129528999328,
+      "epoch": 0.5128205128205128,
+      "grad_norm": 2.234375,
+      "learning_rate": 1.743832442725599e-05,
+      "loss": 0.9296504974365234,
+      "mean_token_accuracy": 0.7648407304286957,
+      "num_tokens": 4327983.0,
+      "step": 700
+    },
+    {
+      "entropy": 0.9435758924484253,
+      "epoch": 0.5311355311355311,
+      "grad_norm": 2.046875,
+      "learning_rate": 1.7236844173760286e-05,
+      "loss": 0.9163207244873047,
+      "mean_token_accuracy": 0.7653989827632904,
+      "num_tokens": 4482610.0,
+      "step": 725
+    },
+    {
+      "entropy": 0.9563624393939972,
+      "epoch": 0.5494505494505495,
+      "grad_norm": 2.296875,
+      "learning_rate": 1.7028998000152368e-05,
+      "loss": 0.942515869140625,
+      "mean_token_accuracy": 0.7614016819000244,
+      "num_tokens": 4638096.0,
+      "step": 750
+    },
+    {
+      "entropy": 0.9195479357242584,
+      "epoch": 0.5677655677655677,
+      "grad_norm": 2.3125,
+      "learning_rate": 1.681496873918434e-05,
+      "loss": 0.8994998931884766,
+      "mean_token_accuracy": 0.7696708023548127,
+      "num_tokens": 4796965.0,
+      "step": 775
+    },
+    {
+      "entropy": 0.9382291078567505,
+      "epoch": 0.5860805860805861,
+      "grad_norm": 2.453125,
+      "learning_rate": 1.659494466258685e-05,
+      "loss": 0.9111621856689454,
+      "mean_token_accuracy": 0.7701178181171418,
+      "num_tokens": 4954572.0,
+      "step": 800
+    },
+    {
+      "epoch": 0.5860805860805861,
+      "eval_entropy": 0.9060760368075635,
+      "eval_loss": 0.9103703498840332,
+      "eval_mean_token_accuracy": 0.7709767441782687,
+      "eval_num_tokens": 4954572.0,
+      "eval_runtime": 8.0596,
+      "eval_samples_per_second": 285.248,
+      "eval_steps_per_second": 17.867,
+      "step": 800
+    },
+    {
+      "entropy": 0.9392394709587097,
+      "epoch": 0.6043956043956044,
+      "grad_norm": 2.34375,
+      "learning_rate": 1.6369119315455067e-05,
+      "loss": 0.9231902313232422,
+      "mean_token_accuracy": 0.7648055994510651,
+      "num_tokens": 5107234.0,
+      "step": 825
+    },
+    {
+      "entropy": 0.9370434725284577,
+      "epoch": 0.6227106227106227,
+      "grad_norm": 2.0625,
+      "learning_rate": 1.613769134599595e-05,
+      "loss": 0.9138022613525391,
+      "mean_token_accuracy": 0.7679059386253357,
+      "num_tokens": 5256697.0,
+      "step": 850
+    },
+    {
+      "entropy": 0.9104315054416656,
+      "epoch": 0.6410256410256411,
+      "grad_norm": 2.15625,
+      "learning_rate": 1.590086433078652e-05,
+      "loss": 0.8872809600830078,
+      "mean_token_accuracy": 0.7746103525161743,
+      "num_tokens": 5406816.0,
+      "step": 875
+    },
+    {
+      "entropy": 0.9144675505161285,
+      "epoch": 0.6593406593406593,
+      "grad_norm": 2.125,
+      "learning_rate": 1.5658846595696868e-05,
+      "loss": 0.8894432830810547,
+      "mean_token_accuracy": 0.7708098363876342,
+      "num_tokens": 5558294.0,
+      "step": 900
+    },
+    {
+      "entropy": 0.9206836307048798,
+      "epoch": 0.6776556776556777,
+      "grad_norm": 2.125,
+      "learning_rate": 1.541185103263545e-05,
+      "loss": 0.8961500549316406,
+      "mean_token_accuracy": 0.771810313463211,
+      "num_tokens": 5712111.0,
+      "step": 925
+    },
+    {
+      "entropy": 0.9454787099361419,
+      "epoch": 0.6959706959706959,
+      "grad_norm": 2.015625,
+      "learning_rate": 1.5160094912277816e-05,
+      "loss": 0.9270650482177735,
+      "mean_token_accuracy": 0.7659087121486664,
+      "num_tokens": 5867502.0,
+      "step": 950
+    },
+    {
+      "entropy": 0.9311832976341248,
+      "epoch": 0.7142857142857143,
+      "grad_norm": 2.3125,
+      "learning_rate": 1.4903799692943575e-05,
+      "loss": 0.9082266235351563,
+      "mean_token_accuracy": 0.7696301078796387,
+      "num_tokens": 6017997.0,
+      "step": 975
+    },
+    {
+      "entropy": 0.9084031569957733,
+      "epoch": 0.7326007326007326,
+      "grad_norm": 2.265625,
+      "learning_rate": 1.4643190825789646e-05,
+      "loss": 0.8856626129150391,
+      "mean_token_accuracy": 0.7754030025005341,
+      "num_tokens": 6170646.0,
+      "step": 1000
+    },
+    {
+      "epoch": 0.7326007326007326,
+      "eval_entropy": 0.9303645570245054,
+      "eval_loss": 0.9040034413337708,
+      "eval_mean_token_accuracy": 0.7716102877424823,
+      "eval_num_tokens": 6170646.0,
+      "eval_runtime": 8.0677,
+      "eval_samples_per_second": 284.963,
+      "eval_steps_per_second": 17.849,
+      "step": 1000
+    },
+    {
+      "entropy": 0.9288157105445862,
+      "epoch": 0.7509157509157509,
+      "grad_norm": 2.125,
+      "learning_rate": 1.4378497556491212e-05,
+      "loss": 0.9045313262939453,
+      "mean_token_accuracy": 0.7689747631549835,
+      "num_tokens": 6325950.0,
+      "step": 1025
+    },
+    {
+      "entropy": 0.9248845767974854,
+      "epoch": 0.7692307692307693,
+      "grad_norm": 2.296875,
+      "learning_rate": 1.410995272358482e-05,
+      "loss": 0.9021368408203125,
+      "mean_token_accuracy": 0.7702593433856965,
+      "num_tokens": 6475837.0,
+      "step": 1050
+    },
+    {
+      "entropy": 0.9433277988433838,
+      "epoch": 0.7875457875457875,
+      "grad_norm": 2.03125,
+      "learning_rate": 1.383779255365097e-05,
+      "loss": 0.9240187072753906,
+      "mean_token_accuracy": 0.7659425389766693,
+      "num_tokens": 6625842.0,
+      "step": 1075
+    },
+    {
+      "entropy": 0.9574247586727143,
+      "epoch": 0.8058608058608059,
+      "grad_norm": 2.265625,
+      "learning_rate": 1.356225645351645e-05,
+      "loss": 0.9296712493896484,
+      "mean_token_accuracy": 0.7633269345760345,
+      "num_tokens": 6777307.0,
+      "step": 1100
+    },
+    {
+      "entropy": 0.9137916600704193,
+      "epoch": 0.8241758241758241,
+      "grad_norm": 2.234375,
+      "learning_rate": 1.3283586799659083e-05,
+      "loss": 0.8911846160888672,
+      "mean_token_accuracy": 0.7715334737300873,
+      "num_tokens": 6932314.0,
+      "step": 1125
+    },
+    {
+      "entropy": 0.9397987961769104,
+      "epoch": 0.8424908424908425,
+      "grad_norm": 2.171875,
+      "learning_rate": 1.3002028725000296e-05,
+      "loss": 0.9209308624267578,
+      "mean_token_accuracy": 0.7674053549766541,
+      "num_tokens": 7082476.0,
+      "step": 1150
+    },
+    {
+      "entropy": 0.9472148299217225,
+      "epoch": 0.8608058608058609,
+      "grad_norm": 2.15625,
+      "learning_rate": 1.2717829903272889e-05,
+      "loss": 0.9266857147216797,
+      "mean_token_accuracy": 0.7626336240768432,
+      "num_tokens": 7239624.0,
+      "step": 1175
+    },
+    {
+      "entropy": 0.9480014276504517,
+      "epoch": 0.8791208791208791,
+      "grad_norm": 2.3125,
+      "learning_rate": 1.243124033115382e-05,
+      "loss": 0.9297947692871094,
+      "mean_token_accuracy": 0.762817633152008,
+      "num_tokens": 7394047.0,
+      "step": 1200
+    },
+    {
+      "epoch": 0.8791208791208791,
+      "eval_entropy": 0.9203641795449786,
+      "eval_loss": 0.8990768790245056,
+      "eval_mean_token_accuracy": 0.7723050812880198,
+      "eval_num_tokens": 7394047.0,
+      "eval_runtime": 8.072,
+      "eval_samples_per_second": 284.813,
+      "eval_steps_per_second": 17.84,
+      "step": 1200
+    },
+    {
+      "entropy": 0.9616302692890167,
+      "epoch": 0.8974358974358975,
+      "grad_norm": 2.640625,
+      "learning_rate": 1.2142512108353583e-05,
+      "loss": 0.9401410675048828,
+      "mean_token_accuracy": 0.7661498367786408,
+      "num_tokens": 7548323.0,
+      "step": 1225
+    },
+    {
+      "entropy": 0.9228850102424622,
+      "epoch": 0.9157509157509157,
+      "grad_norm": 2.21875,
+      "learning_rate": 1.1851899215855653e-05,
+      "loss": 0.8985552215576171,
+      "mean_token_accuracy": 0.7717998087406158,
+      "num_tokens": 7700481.0,
+      "step": 1250
+    },
+    {
+      "entropy": 0.8970145070552826,
+      "epoch": 0.9340659340659341,
+      "grad_norm": 2.296875,
+      "learning_rate": 1.1559657292501044e-05,
+      "loss": 0.87716064453125,
+      "mean_token_accuracy": 0.7763631820678711,
+      "num_tokens": 7852495.0,
+      "step": 1275
+    },
+    {
+      "entropy": 0.9301983499526978,
+      "epoch": 0.9523809523809523,
+      "grad_norm": 2.09375,
+      "learning_rate": 1.1266043410114557e-05,
+      "loss": 0.907692642211914,
+      "mean_token_accuracy": 0.7715549063682556,
+      "num_tokens": 8001324.0,
+      "step": 1300
+    },
+    {
+      "entropy": 0.91446182847023,
+      "epoch": 0.9706959706959707,
+      "grad_norm": 2.359375,
+      "learning_rate": 1.0971315847370463e-05,
+      "loss": 0.8939350891113281,
+      "mean_token_accuracy": 0.7720365560054779,
+      "num_tokens": 8161385.0,
+      "step": 1325
+    },
+    {
+      "entropy": 0.9196098625659943,
+      "epoch": 0.989010989010989,
+      "grad_norm": 2.109375,
+      "learning_rate": 1.06757338625966e-05,
+      "loss": 0.89285888671875,
+      "mean_token_accuracy": 0.7721686267852783,
+      "num_tokens": 8315619.0,
+      "step": 1350
+    },
+    {
+      "entropy": 0.9019503581523896,
+      "epoch": 1.0073260073260073,
+      "grad_norm": 1.8828125,
+      "learning_rate": 1.0379557465716696e-05,
+      "loss": 0.853975830078125,
+      "mean_token_accuracy": 0.7786379647254944,
+      "num_tokens": 8474116.0,
+      "step": 1375
+    },
+    {
+      "entropy": 0.8720257365703583,
+      "epoch": 1.0256410256410255,
+      "grad_norm": 2.140625,
+      "learning_rate": 1.0083047189531566e-05,
+      "loss": 0.8370319366455078,
+      "mean_token_accuracy": 0.7838277363777161,
+      "num_tokens": 8633284.0,
+      "step": 1400
+    },
+    {
+      "epoch": 1.0256410256410255,
+      "eval_entropy": 0.8584034711950355,
+      "eval_loss": 0.9003849029541016,
+      "eval_mean_token_accuracy": 0.7730862986710336,
+      "eval_num_tokens": 8633284.0,
+      "eval_runtime": 8.0586,
+      "eval_samples_per_second": 285.285,
+      "eval_steps_per_second": 17.869,
+      "step": 1400
+    },
+    {
+      "entropy": 0.8384023177623748,
+      "epoch": 1.043956043956044,
+      "grad_norm": 2.15625,
+      "learning_rate": 9.786463860540352e-06,
+      "loss": 0.7985468292236328,
+      "mean_token_accuracy": 0.7922709083557129,
+      "num_tokens": 8785152.0,
+      "step": 1425
+    },
+    {
+      "entropy": 0.848993946313858,
+      "epoch": 1.0622710622710623,
+      "grad_norm": 2.234375,
+      "learning_rate": 9.49006836950339e-06,
+      "loss": 0.8184638977050781,
+      "mean_token_accuracy": 0.7883424413204193,
+      "num_tokens": 8936429.0,
+      "step": 1450
+    },
+    {
+      "entropy": 0.8467873334884644,
+      "epoch": 1.0805860805860805,
+      "grad_norm": 2.140625,
+      "learning_rate": 9.194121441948596e-06,
+      "loss": 0.8056396484375,
+      "mean_token_accuracy": 0.7903980255126953,
+      "num_tokens": 9094633.0,
+      "step": 1475
+    },
+    {
+      "entropy": 0.8402684187889099,
+      "epoch": 1.098901098901099,
+      "grad_norm": 2.109375,
+      "learning_rate": 8.898883408823124e-06,
+      "loss": 0.8085630798339843,
+      "mean_token_accuracy": 0.7898635935783386,
+      "num_tokens": 9246323.0,
+      "step": 1500
+    },
+    {
+      "entropy": 0.8303942859172821,
+      "epoch": 1.1172161172161172,
+      "grad_norm": 2.0625,
+      "learning_rate": 8.604613977492212e-06,
+      "loss": 0.7932289123535157,
+      "mean_token_accuracy": 0.7939286470413208,
+      "num_tokens": 9395722.0,
+      "step": 1525
+    },
+    {
+      "entropy": 0.8481007432937622,
+      "epoch": 1.1355311355311355,
+      "grad_norm": 2.109375,
+      "learning_rate": 8.3115720032865e-06,
+      "loss": 0.8194800567626953,
+      "mean_token_accuracy": 0.7893430662155151,
+      "num_tokens": 9547484.0,
+      "step": 1550
+    },
+    {
+      "entropy": 0.8354680657386779,
+      "epoch": 1.1538461538461537,
+      "grad_norm": 2.265625,
+      "learning_rate": 8.02001526179883e-06,
+      "loss": 0.8005447387695312,
+      "mean_token_accuracy": 0.7913288366794586,
+      "num_tokens": 9699959.0,
+      "step": 1575
+    },
+    {
+      "entropy": 0.8411949121952057,
+      "epoch": 1.1721611721611722,
+      "grad_norm": 2.125,
+      "learning_rate": 7.730200222130932e-06,
+      "loss": 0.810956039428711,
+      "mean_token_accuracy": 0.7901583468914032,
+      "num_tokens": 9851149.0,
+      "step": 1600
+    },
+    {
+      "epoch": 1.1721611721611722,
+      "eval_entropy": 0.8544707364506192,
+      "eval_loss": 0.8998146057128906,
+      "eval_mean_token_accuracy": 0.7731748426126109,
+      "eval_num_tokens": 9851149.0,
+      "eval_runtime": 8.0715,
+      "eval_samples_per_second": 284.828,
+      "eval_steps_per_second": 17.84,
+      "step": 1600
+    },
+    {
+      "entropy": 0.8444626975059509,
+      "epoch": 1.1904761904761905,
+      "grad_norm": 2.328125,
+      "learning_rate": 7.442381821289248e-06,
+      "loss": 0.8108631896972657,
+      "mean_token_accuracy": 0.7912653696537018,
+      "num_tokens": 9998793.0,
+      "step": 1625
+    },
+    {
+      "entropy": 0.8223755669593811,
+      "epoch": 1.2087912087912087,
+      "grad_norm": 2.09375,
+      "learning_rate": 7.1568132399285794e-06,
+      "loss": 0.7869499969482422,
+      "mean_token_accuracy": 0.7936726534366607,
+      "num_tokens": 10152752.0,
+      "step": 1650
+    },
+    {
+      "entropy": 0.8130457592010498,
+      "epoch": 1.2271062271062272,
+      "grad_norm": 1.875,
+      "learning_rate": 6.8737456796406594e-06,
+      "loss": 0.78310546875,
+      "mean_token_accuracy": 0.7966435635089875,
+      "num_tokens": 10308001.0,
+      "step": 1675
+    },
+    {
+      "entropy": 0.8471330416202545,
+      "epoch": 1.2454212454212454,
+      "grad_norm": 2.046875,
+      "learning_rate": 6.593428141983595e-06,
+      "loss": 0.8101668548583985,
+      "mean_token_accuracy": 0.7890221488475799,
+      "num_tokens": 10459655.0,
+      "step": 1700
+    },
+    {
+      "entropy": 0.8247973656654358,
+      "epoch": 1.2637362637362637,
+      "grad_norm": 2.1875,
+      "learning_rate": 6.316107209446662e-06,
+      "loss": 0.796153564453125,
+      "mean_token_accuracy": 0.7926836657524109,
+      "num_tokens": 10615183.0,
+      "step": 1725
+    },
+    {
+      "entropy": 0.8476881444454193,
+      "epoch": 1.282051282051282,
+      "grad_norm": 2.1875,
+      "learning_rate": 6.042026828542942e-06,
+      "loss": 0.8173642730712891,
+      "mean_token_accuracy": 0.7872111082077027,
+      "num_tokens": 10774019.0,
+      "step": 1750
+    },
+    {
+      "entropy": 0.8313502633571624,
+      "epoch": 1.3003663003663004,
+      "grad_norm": 2.0,
+      "learning_rate": 5.7714280952207955e-06,
+      "loss": 0.8023670959472656,
+      "mean_token_accuracy": 0.7926879894733428,
+      "num_tokens": 10926453.0,
+      "step": 1775
+    },
+    {
+      "entropy": 0.8281714594364167,
+      "epoch": 1.3186813186813187,
+      "grad_norm": 2.296875,
+      "learning_rate": 5.504549042782779e-06,
+      "loss": 0.7919468688964844,
+      "mean_token_accuracy": 0.7943351435661316,
+      "num_tokens": 11076817.0,
+      "step": 1800
+    },
+    {
+      "epoch": 1.3186813186813187,
+      "eval_entropy": 0.8518242807024055,
+      "eval_loss": 0.8994762897491455,
+      "eval_mean_token_accuracy": 0.7733377065095637,
+      "eval_num_tokens": 11076817.0,
+      "eval_runtime": 8.0691,
+      "eval_samples_per_second": 284.912,
+      "eval_steps_per_second": 17.846,
+      "step": 1800
+    },
+    {
+      "entropy": 0.8253132402896881,
+      "epoch": 1.3369963369963371,
+      "grad_norm": 2.09375,
+      "learning_rate": 5.241624432498673e-06,
+      "loss": 0.7934477996826171,
+      "mean_token_accuracy": 0.7905769181251526,
+      "num_tokens": 11235359.0,
+      "step": 1825
+    },
+    {
+      "entropy": 0.8527077162265777,
+      "epoch": 1.3553113553113554,
+      "grad_norm": 2.25,
+      "learning_rate": 4.982885547096754e-06,
+      "loss": 0.8256228637695312,
+      "mean_token_accuracy": 0.7879673659801483,
+      "num_tokens": 11385249.0,
+      "step": 1850
+    },
+    {
+      "entropy": 0.8239582288265228,
+      "epoch": 1.3736263736263736,
+      "grad_norm": 2.09375,
+      "learning_rate": 4.728559987314975e-06,
+      "loss": 0.7923411560058594,
+      "mean_token_accuracy": 0.7940970385074615,
+      "num_tokens": 11539970.0,
+      "step": 1875
+    },
+    {
+      "entropy": 0.8423762857913971,
+      "epoch": 1.3919413919413919,
+      "grad_norm": 2.1875,
+      "learning_rate": 4.478871471691031e-06,
+      "loss": 0.8075601196289063,
+      "mean_token_accuracy": 0.7895544397830964,
+      "num_tokens": 11694684.0,
+      "step": 1900
+    },
+    {
+      "entropy": 0.8148546552658081,
+      "epoch": 1.4102564102564101,
+      "grad_norm": 2.125,
+      "learning_rate": 4.234039639767406e-06,
+      "loss": 0.7813822937011718,
+      "mean_token_accuracy": 0.7967073798179627,
+      "num_tokens": 11850074.0,
+      "step": 1925
+    },
+    {
+      "entropy": 0.840382571220398,
+      "epoch": 1.4285714285714286,
+      "grad_norm": 2.09375,
+      "learning_rate": 3.994279858884551e-06,
+      "loss": 0.8069898986816406,
+      "mean_token_accuracy": 0.7921997106075287,
+      "num_tokens": 12004729.0,
+      "step": 1950
+    },
+    {
+      "entropy": 0.8522861528396607,
+      "epoch": 1.4468864468864469,
+      "grad_norm": 2.03125,
+      "learning_rate": 3.7598030347321e-06,
+      "loss": 0.8208354187011718,
+      "mean_token_accuracy": 0.789222640991211,
+      "num_tokens": 12157974.0,
+      "step": 1975
+    },
+    {
+      "entropy": 0.8414350926876069,
+      "epoch": 1.4652014652014653,
+      "grad_norm": 2.078125,
+      "learning_rate": 3.530815425824797e-06,
+      "loss": 0.8093458557128906,
+      "mean_token_accuracy": 0.7892864966392517,
+      "num_tokens": 12306150.0,
+      "step": 2000
+    },
+    {
+      "epoch": 1.4652014652014653,
+      "eval_entropy": 0.8502832547658019,
+      "eval_loss": 0.8990073204040527,
+      "eval_mean_token_accuracy": 0.7732960196832815,
+      "eval_num_tokens": 12306150.0,
+      "eval_runtime": 8.068,
+      "eval_samples_per_second": 284.952,
+      "eval_steps_per_second": 17.848,
+      "step": 2000
+    }
+  ],
+  "logging_steps": 25,
+  "max_steps": 2730,
+  "num_input_tokens_seen": 0,
+  "num_train_epochs": 2,
+  "save_steps": 200,
+  "stateful_callbacks": {
+    "TrainerControl": {
+      "args": {
+        "should_epoch_stop": false,
+        "should_evaluate": false,
+        "should_log": false,
+        "should_save": true,
+        "should_training_stop": false
+      },
+      "attributes": {}
+    }
+  },
+  "total_flos": 4.976436110229043e+16,
+  "train_batch_size": 16,
+  "trial_name": null,
+  "trial_params": null
+}

checkpoint-2000/training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1f3eec82a82cc2cb4f2c124367cd9fb480a301c5cdfda379c7339e6e590e5bd3
+size 5585

checkpoint-2600/chat_template.jinja

@@ -0,0 +1,64 @@
+{{- bos_token -}}
+{%- set keep_past_thinking = keep_past_thinking | default(false) -%}
+{%- set ns = namespace(system_prompt="") -%}
+{%- if messages[0]["role"] == "system" -%}
+    {%- set sys_content = messages[0]["content"] -%}
+    {%- if sys_content is not string -%}
+        {%- for item in sys_content -%}
+            {%- if item["type"] == "text" -%}
+                {%- set ns.system_prompt = ns.system_prompt + item["text"] -%}
+            {%- endif -%}
+        {%- endfor -%}
+    {%- else -%}
+        {%- set ns.system_prompt = sys_content -%}
+    {%- endif -%}
+    {%- set messages = messages[1:] -%}
+{%- endif -%}
+{%- if tools -%}
+    {%- set ns.system_prompt = ns.system_prompt + ("\n" if ns.system_prompt else "") + "List of tools: [" -%}
+    {%- for tool in tools -%}
+        {%- if tool is not string -%}
+            {%- set tool = tool | tojson -%}
+        {%- endif -%}
+        {%- set ns.system_prompt = ns.system_prompt + tool -%}
+        {%- if not loop.last -%}
+            {%- set ns.system_prompt = ns.system_prompt + ", " -%}
+        {%- endif -%}
+    {%- endfor -%}
+    {%- set ns.system_prompt = ns.system_prompt + "]" -%}
+{%- endif -%}
+{%- if ns.system_prompt -%}
+    {{- "<|im_start|>system\n" + ns.system_prompt + "<|im_end|>\n" -}}
+{%- endif -%}
+{%- set ns.last_assistant_index = -1 -%}
+{%- for message in messages -%}
+    {%- if message["role"] == "assistant" -%}
+        {%- set ns.last_assistant_index = loop.index0 -%}
+    {%- endif -%}
+{%- endfor -%}
+{%- for message in messages -%}
+    {{- "<|im_start|>" + message["role"] + "\n" -}}
+    {%- set content = message["content"] -%}
+    {%- if content is not string -%}
+        {%- set ns.content = "" -%}
+        {%- for item in content -%}
+            {%- if item["type"] == "image" -%}
+                {%- set ns.content = ns.content + "<image>" -%}
+            {%- elif item["type"] == "text" -%}
+                {%- set ns.content = ns.content + item["text"] -%}
+            {%- else -%}
+                {%- set ns.content = ns.content + item | tojson -%}
+            {%- endif -%}
+        {%- endfor -%}
+        {%- set content = ns.content -%}
+    {%- endif -%}
+    {%- if message["role"] == "assistant" and not keep_past_thinking and loop.index0 != ns.last_assistant_index -%}
+        {%- if "</think>" in content -%}
+            {%- set content = content.split("</think>")[-1] | trim -%}
+        {%- endif -%}
+    {%- endif -%}
+    {{- content + "<|im_end|>\n" -}}
+{%- endfor -%}
+{%- if add_generation_prompt -%}
+    {{- "<|im_start|>assistant\n" -}}
+{%- endif -%}

checkpoint-2600/config.json

@@ -0,0 +1,60 @@
+{
+  "architectures": [
+    "Lfm2ForCausalLM"
+  ],
+  "block_auto_adjust_ff_dim": true,
+  "block_dim": 1024,
+  "block_ffn_dim_multiplier": 1.0,
+  "block_mlp_init_scale": 1.0,
+  "block_multiple_of": 256,
+  "block_norm_eps": 1e-05,
+  "block_out_init_scale": 1.0,
+  "block_use_swiglu": true,
+  "block_use_xavier_init": true,
+  "bos_token_id": 1,
+  "conv_L_cache": 3,
+  "conv_bias": false,
+  "conv_dim": 1024,
+  "conv_use_xavier_init": true,
+  "dtype": "bfloat16",
+  "eos_token_id": 7,
+  "full_attn_idxs": null,
+  "hidden_size": 1024,
+  "initializer_range": 0.02,
+  "intermediate_size": 6656,
+  "layer_types": [
+    "conv",
+    "conv",
+    "full_attention",
+    "conv",
+    "conv",
+    "full_attention",
+    "conv",
+    "conv",
+    "full_attention",
+    "conv",
+    "full_attention",
+    "conv",
+    "full_attention",
+    "conv",
+    "full_attention",
+    "conv"
+  ],
+  "max_position_embeddings": 128000,
+  "model_type": "lfm2",
+  "norm_eps": 1e-05,
+  "num_attention_heads": 16,
+  "num_heads": 16,
+  "num_hidden_layers": 16,
+  "num_key_value_heads": 8,
+  "pad_token_id": 0,
+  "rope_parameters": {
+    "rope_theta": 1000000.0,
+    "rope_type": "default"
+  },
+  "tie_word_embeddings": true,
+  "transformers_version": "5.5.0",
+  "use_cache": false,
+  "use_pos_enc": true,
+  "vocab_size": 65536
+}

checkpoint-2600/generation_config.json

@@ -0,0 +1,9 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 1,
+  "eos_token_id": [
+    7
+  ],
+  "pad_token_id": 0,
+  "transformers_version": "5.5.0"
+}

checkpoint-2600/model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d16491df657d762fc4c37c146ebf453d64f33e2c81bffed3f2dbdf792f919eba
+size 708984464

checkpoint-2600/optimizer.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:96e640aa81ff1ae64ec102ed6ecbf2d8c493232992cc5ff3977426ded723ce68
+size 1418061003

checkpoint-2600/rng_state.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9159128caf9279731957c10866cdf0663e56c47f6aae1443f23ed441fd8fb6f8
+size 14645

checkpoint-2600/scheduler.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ca804db8dc9667741ad7b80c79f0a1e49ed823c643bbaa675b0830b3a56d2e2b
+size 1465

checkpoint-2600/tokenizer_config.json

@@ -0,0 +1,21 @@
+{
+  "backend": "tokenizers",
+  "bos_token": "<|startoftext|>",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|im_end|>",
+  "extra_special_tokens": [],
+  "is_local": false,
+  "legacy": false,
+  "model_input_names": [
+    "input_ids",
+    "attention_mask"
+  ],
+  "model_max_length": 1000000000000000019884624838656,
+  "model_specific_special_tokens": {},
+  "pad_token": "<|pad|>",
+  "sp_model_kwargs": {},
+  "spaces_between_special_tokens": false,
+  "tokenizer_class": "TokenizersBackend",
+  "use_default_system_prompt": false,
+  "use_fast": true
+}

checkpoint-2600/trainer_state.json

@@ -0,0 +1,1217 @@
+{
+  "best_global_step": 2000,
+  "best_metric": 0.8990073204040527,
+  "best_model_checkpoint": "lfm2.5-350m-balanced/checkpoint-2000",
+  "epoch": 1.9047619047619047,
+  "eval_steps": 200,
+  "global_step": 2600,
+  "is_hyper_param_search": false,
+  "is_local_process_zero": true,
+  "is_world_process_zero": true,
+  "log_history": [
+    {
+      "entropy": 1.20119291305542,
+      "epoch": 0.018315018315018316,
+      "grad_norm": 10.4375,
+      "learning_rate": 5.853658536585366e-06,
+      "loss": 2.8065673828125,
+      "mean_token_accuracy": 0.57585333943367,
+      "num_tokens": 148430.0,
+      "step": 25
+    },
+    {
+      "entropy": 1.278236083984375,
+      "epoch": 0.03663003663003663,
+      "grad_norm": 4.125,
+      "learning_rate": 1.1951219512195123e-05,
+      "loss": 1.506644287109375,
+      "mean_token_accuracy": 0.7009313237667084,
+      "num_tokens": 301448.0,
+      "step": 50
+    },
+    {
+      "entropy": 1.0903868675231934,
+      "epoch": 0.054945054945054944,
+      "grad_norm": 2.609375,
+      "learning_rate": 1.804878048780488e-05,
+      "loss": 1.0758729553222657,
+      "mean_token_accuracy": 0.7404501688480377,
+      "num_tokens": 458432.0,
+      "step": 75
+    },
+    {
+      "entropy": 1.0081040942668915,
+      "epoch": 0.07326007326007326,
+      "grad_norm": 2.4375,
+      "learning_rate": 1.9997966159416884e-05,
+      "loss": 0.9917147827148437,
+      "mean_token_accuracy": 0.7522138917446136,
+      "num_tokens": 616048.0,
+      "step": 100
+    },
+    {
+      "entropy": 0.9854830229282379,
+      "epoch": 0.09157509157509157,
+      "grad_norm": 2.46875,
+      "learning_rate": 1.99875879788391e-05,
+      "loss": 0.9723490905761719,
+      "mean_token_accuracy": 0.7565386164188385,
+      "num_tokens": 772466.0,
+      "step": 125
+    },
+    {
+      "entropy": 0.9745642852783203,
+      "epoch": 0.10989010989010989,
+      "grad_norm": 2.25,
+      "learning_rate": 1.996842417490666e-05,
+      "loss": 0.9625119018554688,
+      "mean_token_accuracy": 0.760125788450241,
+      "num_tokens": 930723.0,
+      "step": 150
+    },
+    {
+      "entropy": 0.9700245261192322,
+      "epoch": 0.1282051282051282,
+      "grad_norm": 2.234375,
+      "learning_rate": 1.99404916051395e-05,
+      "loss": 0.9550397491455078,
+      "mean_token_accuracy": 0.7578657758235932,
+      "num_tokens": 1091160.0,
+      "step": 175
+    },
+    {
+      "entropy": 0.9669338822364807,
+      "epoch": 0.14652014652014653,
+      "grad_norm": 2.359375,
+      "learning_rate": 1.9903814840538914e-05,
+      "loss": 0.9478360748291016,
+      "mean_token_accuracy": 0.7623070132732391,
+      "num_tokens": 1245781.0,
+      "step": 200
+    },
+    {
+      "epoch": 0.14652014652014653,
+      "eval_entropy": 0.9581337045464251,
+      "eval_loss": 0.9430390000343323,
+      "eval_mean_token_accuracy": 0.7639545847972234,
+      "eval_num_tokens": 1245781.0,
+      "eval_runtime": 8.0902,
+      "eval_samples_per_second": 284.171,
+      "eval_steps_per_second": 17.799,
+      "step": 200
+    },
+    {
+      "entropy": 0.9653169775009155,
+      "epoch": 0.16483516483516483,
+      "grad_norm": 2.640625,
+      "learning_rate": 1.9858426143973623e-05,
+      "loss": 0.9534239959716797,
+      "mean_token_accuracy": 0.7631052339076996,
+      "num_tokens": 1395650.0,
+      "step": 225
+    },
+    {
+      "entropy": 0.962496577501297,
+      "epoch": 0.18315018315018314,
+      "grad_norm": 2.234375,
+      "learning_rate": 1.9804365441799555e-05,
+      "loss": 0.9441399383544922,
+      "mean_token_accuracy": 0.7629920542240143,
+      "num_tokens": 1553420.0,
+      "step": 250
+    },
+    {
+      "entropy": 0.9607322406768799,
+      "epoch": 0.20146520146520147,
+      "grad_norm": 2.46875,
+      "learning_rate": 1.9741680288738495e-05,
+      "loss": 0.939505615234375,
+      "mean_token_accuracy": 0.7633682239055634,
+      "num_tokens": 1708083.0,
+      "step": 275
+    },
+    {
+      "entropy": 0.9339856076240539,
+      "epoch": 0.21978021978021978,
+      "grad_norm": 2.203125,
+      "learning_rate": 1.967042582604635e-05,
+      "loss": 0.9153070831298828,
+      "mean_token_accuracy": 0.767605767250061,
+      "num_tokens": 1862545.0,
+      "step": 300
+    },
+    {
+      "entropy": 0.9721270000934601,
+      "epoch": 0.23809523809523808,
+      "grad_norm": 2.265625,
+      "learning_rate": 1.9590664733007947e-05,
+      "loss": 0.959111328125,
+      "mean_token_accuracy": 0.7571033167839051,
+      "num_tokens": 2017595.0,
+      "step": 325
+    },
+    {
+      "entropy": 0.961563161611557,
+      "epoch": 0.2564102564102564,
+      "grad_norm": 2.3125,
+      "learning_rate": 1.9502467171800902e-05,
+      "loss": 0.9458185577392578,
+      "mean_token_accuracy": 0.7628096437454224,
+      "num_tokens": 2166320.0,
+      "step": 350
+    },
+    {
+      "entropy": 0.9491504514217377,
+      "epoch": 0.27472527472527475,
+      "grad_norm": 2.34375,
+      "learning_rate": 1.940591072577719e-05,
+      "loss": 0.9269615173339844,
+      "mean_token_accuracy": 0.7635479891300201,
+      "num_tokens": 2325678.0,
+      "step": 375
+    },
+    {
+      "entropy": 0.9450718343257904,
+      "epoch": 0.29304029304029305,
+      "grad_norm": 2.28125,
+      "learning_rate": 1.930108033121663e-05,
+      "loss": 0.9313880920410156,
+      "mean_token_accuracy": 0.7636011290550232,
+      "num_tokens": 2481053.0,
+      "step": 400
+    },
+    {
+      "epoch": 0.29304029304029305,
+      "eval_entropy": 0.9506676958666908,
+      "eval_loss": 0.9265689849853516,
+      "eval_mean_token_accuracy": 0.7673345183332761,
+      "eval_num_tokens": 2481053.0,
+      "eval_runtime": 8.0719,
+      "eval_samples_per_second": 284.816,
+      "eval_steps_per_second": 17.84,
+      "step": 400
+    },
+    {
+      "entropy": 0.9616206979751587,
+      "epoch": 0.31135531135531136,
+      "grad_norm": 2.28125,
+      "learning_rate": 1.9188068202612317e-05,
+      "loss": 0.9455902862548828,
+      "mean_token_accuracy": 0.7630793690681458,
+      "num_tokens": 2633845.0,
+      "step": 425
+    },
+    {
+      "entropy": 0.9392594003677368,
+      "epoch": 0.32967032967032966,
+      "grad_norm": 2.3125,
+      "learning_rate": 1.906697375155377e-05,
+      "loss": 0.9156442260742188,
+      "mean_token_accuracy": 0.7659864926338196,
+      "num_tokens": 2790745.0,
+      "step": 450
+    },
+    {
+      "entropy": 0.9245667338371277,
+      "epoch": 0.34798534798534797,
+      "grad_norm": 2.203125,
+      "learning_rate": 1.8937903499279104e-05,
+      "loss": 0.903341064453125,
+      "mean_token_accuracy": 0.7713142216205597,
+      "num_tokens": 2943444.0,
+      "step": 475
+    },
+    {
+      "entropy": 0.912586669921875,
+      "epoch": 0.3663003663003663,
+      "grad_norm": 2.390625,
+      "learning_rate": 1.880097098297319e-05,
+      "loss": 0.8922891235351562,
+      "mean_token_accuracy": 0.771296215057373,
+      "num_tokens": 3098959.0,
+      "step": 500
+    },
+    {
+      "entropy": 0.9415633583068848,
+      "epoch": 0.38461538461538464,
+      "grad_norm": 2.15625,
+      "learning_rate": 1.8656296655894205e-05,
+      "loss": 0.9213317108154296,
+      "mean_token_accuracy": 0.7668669641017913,
+      "num_tokens": 3248382.0,
+      "step": 525
+    },
+    {
+      "entropy": 0.9617620420455932,
+      "epoch": 0.40293040293040294,
+      "grad_norm": 2.109375,
+      "learning_rate": 1.8504007781416385e-05,
+      "loss": 0.9434718322753907,
+      "mean_token_accuracy": 0.7617218482494355,
+      "num_tokens": 3409803.0,
+      "step": 550
+    },
+    {
+      "entropy": 0.9193656361103058,
+      "epoch": 0.42124542124542125,
+      "grad_norm": 2.0625,
+      "learning_rate": 1.8344238321082316e-05,
+      "loss": 0.901815185546875,
+      "mean_token_accuracy": 0.7707051420211792,
+      "num_tokens": 3559416.0,
+      "step": 575
+    },
+    {
+      "entropy": 0.9518638348579407,
+      "epoch": 0.43956043956043955,
+      "grad_norm": 2.15625,
+      "learning_rate": 1.817712881676307e-05,
+      "loss": 0.9327508544921875,
+      "mean_token_accuracy": 0.7644142961502075,
+      "num_tokens": 3710110.0,
+      "step": 600
+    },
+    {
+      "epoch": 0.43956043956043955,
+      "eval_entropy": 0.9517907314002514,
+      "eval_loss": 0.9172877073287964,
+      "eval_mean_token_accuracy": 0.7684686299827364,
+      "eval_num_tokens": 3710110.0,
+      "eval_runtime": 8.0887,
+      "eval_samples_per_second": 284.222,
+      "eval_steps_per_second": 17.803,
+      "step": 600
+    },
+    {
+      "entropy": 0.9130004584789276,
+      "epoch": 0.45787545787545786,
+      "grad_norm": 2.21875,
+      "learning_rate": 1.8002826267029977e-05,
+      "loss": 0.8941175079345703,
+      "mean_token_accuracy": 0.7705032932758331,
+      "num_tokens": 3862020.0,
+      "step": 625
+    },
+    {
+      "entropy": 0.9252553701400756,
+      "epoch": 0.47619047619047616,
+      "grad_norm": 2.15625,
+      "learning_rate": 1.7821483997846722e-05,
+      "loss": 0.9165068054199219,
+      "mean_token_accuracy": 0.7672811985015869,
+      "num_tokens": 4018231.0,
+      "step": 650
+    },
+    {
+      "entropy": 0.9462221479415893,
+      "epoch": 0.4945054945054945,
+      "grad_norm": 2.125,
+      "learning_rate": 1.763326152769551e-05,
+      "loss": 0.9259709167480469,
+      "mean_token_accuracy": 0.7658140981197357,
+      "num_tokens": 4173058.0,
+      "step": 675
+    },
+    {
+      "entropy": 0.9479129528999328,
+      "epoch": 0.5128205128205128,
+      "grad_norm": 2.234375,
+      "learning_rate": 1.743832442725599e-05,
+      "loss": 0.9296504974365234,
+      "mean_token_accuracy": 0.7648407304286957,
+      "num_tokens": 4327983.0,
+      "step": 700
+    },
+    {
+      "entropy": 0.9435758924484253,
+      "epoch": 0.5311355311355311,
+      "grad_norm": 2.046875,
+      "learning_rate": 1.7236844173760286e-05,
+      "loss": 0.9163207244873047,
+      "mean_token_accuracy": 0.7653989827632904,
+      "num_tokens": 4482610.0,
+      "step": 725
+    },
+    {
+      "entropy": 0.9563624393939972,
+      "epoch": 0.5494505494505495,
+      "grad_norm": 2.296875,
+      "learning_rate": 1.7028998000152368e-05,
+      "loss": 0.942515869140625,
+      "mean_token_accuracy": 0.7614016819000244,
+      "num_tokens": 4638096.0,
+      "step": 750
+    },
+    {
+      "entropy": 0.9195479357242584,
+      "epoch": 0.5677655677655677,
+      "grad_norm": 2.3125,
+      "learning_rate": 1.681496873918434e-05,
+      "loss": 0.8994998931884766,
+      "mean_token_accuracy": 0.7696708023548127,
+      "num_tokens": 4796965.0,
+      "step": 775
+    },
+    {
+      "entropy": 0.9382291078567505,
+      "epoch": 0.5860805860805861,
+      "grad_norm": 2.453125,
+      "learning_rate": 1.659494466258685e-05,
+      "loss": 0.9111621856689454,
+      "mean_token_accuracy": 0.7701178181171418,
+      "num_tokens": 4954572.0,
+      "step": 800
+    },
+    {
+      "epoch": 0.5860805860805861,
+      "eval_entropy": 0.9060760368075635,
+      "eval_loss": 0.9103703498840332,
+      "eval_mean_token_accuracy": 0.7709767441782687,
+      "eval_num_tokens": 4954572.0,
+      "eval_runtime": 8.0596,
+      "eval_samples_per_second": 285.248,
+      "eval_steps_per_second": 17.867,
+      "step": 800
+    },
+    {
+      "entropy": 0.9392394709587097,
+      "epoch": 0.6043956043956044,
+      "grad_norm": 2.34375,
+      "learning_rate": 1.6369119315455067e-05,
+      "loss": 0.9231902313232422,
+      "mean_token_accuracy": 0.7648055994510651,
+      "num_tokens": 5107234.0,
+      "step": 825
+    },
+    {
+      "entropy": 0.9370434725284577,
+      "epoch": 0.6227106227106227,
+      "grad_norm": 2.0625,
+      "learning_rate": 1.613769134599595e-05,
+      "loss": 0.9138022613525391,
+      "mean_token_accuracy": 0.7679059386253357,
+      "num_tokens": 5256697.0,
+      "step": 850
+    },
+    {
+      "entropy": 0.9104315054416656,
+      "epoch": 0.6410256410256411,
+      "grad_norm": 2.15625,
+      "learning_rate": 1.590086433078652e-05,
+      "loss": 0.8872809600830078,
+      "mean_token_accuracy": 0.7746103525161743,
+      "num_tokens": 5406816.0,
+      "step": 875
+    },
+    {
+      "entropy": 0.9144675505161285,
+      "epoch": 0.6593406593406593,
+      "grad_norm": 2.125,
+      "learning_rate": 1.5658846595696868e-05,
+      "loss": 0.8894432830810547,
+      "mean_token_accuracy": 0.7708098363876342,
+      "num_tokens": 5558294.0,
+      "step": 900
+    },
+    {
+      "entropy": 0.9206836307048798,
+      "epoch": 0.6776556776556777,
+      "grad_norm": 2.125,
+      "learning_rate": 1.541185103263545e-05,
+      "loss": 0.8961500549316406,
+      "mean_token_accuracy": 0.771810313463211,
+      "num_tokens": 5712111.0,
+      "step": 925
+    },
+    {
+      "entropy": 0.9454787099361419,
+      "epoch": 0.6959706959706959,
+      "grad_norm": 2.015625,
+      "learning_rate": 1.5160094912277816e-05,
+      "loss": 0.9270650482177735,
+      "mean_token_accuracy": 0.7659087121486664,
+      "num_tokens": 5867502.0,
+      "step": 950
+    },
+    {
+      "entropy": 0.9311832976341248,
+      "epoch": 0.7142857142857143,
+      "grad_norm": 2.3125,
+      "learning_rate": 1.4903799692943575e-05,
+      "loss": 0.9082266235351563,
+      "mean_token_accuracy": 0.7696301078796387,
+      "num_tokens": 6017997.0,
+      "step": 975
+    },
+    {
+      "entropy": 0.9084031569957733,
+      "epoch": 0.7326007326007326,
+      "grad_norm": 2.265625,
+      "learning_rate": 1.4643190825789646e-05,
+      "loss": 0.8856626129150391,
+      "mean_token_accuracy": 0.7754030025005341,
+      "num_tokens": 6170646.0,
+      "step": 1000
+    },
+    {
+      "epoch": 0.7326007326007326,
+      "eval_entropy": 0.9303645570245054,
+      "eval_loss": 0.9040034413337708,
+      "eval_mean_token_accuracy": 0.7716102877424823,
+      "eval_num_tokens": 6170646.0,
+      "eval_runtime": 8.0677,
+      "eval_samples_per_second": 284.963,
+      "eval_steps_per_second": 17.849,
+      "step": 1000
+    },
+    {
+      "entropy": 0.9288157105445862,
+      "epoch": 0.7509157509157509,
+      "grad_norm": 2.125,
+      "learning_rate": 1.4378497556491212e-05,
+      "loss": 0.9045313262939453,
+      "mean_token_accuracy": 0.7689747631549835,
+      "num_tokens": 6325950.0,
+      "step": 1025
+    },
+    {
+      "entropy": 0.9248845767974854,
+      "epoch": 0.7692307692307693,
+      "grad_norm": 2.296875,
+      "learning_rate": 1.410995272358482e-05,
+      "loss": 0.9021368408203125,
+      "mean_token_accuracy": 0.7702593433856965,
+      "num_tokens": 6475837.0,
+      "step": 1050
+    },
+    {
+      "entropy": 0.9433277988433838,
+      "epoch": 0.7875457875457875,
+      "grad_norm": 2.03125,
+      "learning_rate": 1.383779255365097e-05,
+      "loss": 0.9240187072753906,
+      "mean_token_accuracy": 0.7659425389766693,
+      "num_tokens": 6625842.0,
+      "step": 1075
+    },
+    {
+      "entropy": 0.9574247586727143,
+      "epoch": 0.8058608058608059,
+      "grad_norm": 2.265625,
+      "learning_rate": 1.356225645351645e-05,
+      "loss": 0.9296712493896484,
+      "mean_token_accuracy": 0.7633269345760345,
+      "num_tokens": 6777307.0,
+      "step": 1100
+    },
+    {
+      "entropy": 0.9137916600704193,
+      "epoch": 0.8241758241758241,
+      "grad_norm": 2.234375,
+      "learning_rate": 1.3283586799659083e-05,
+      "loss": 0.8911846160888672,
+      "mean_token_accuracy": 0.7715334737300873,
+      "num_tokens": 6932314.0,
+      "step": 1125
+    },
+    {
+      "entropy": 0.9397987961769104,
+      "epoch": 0.8424908424908425,
+      "grad_norm": 2.171875,
+      "learning_rate": 1.3002028725000296e-05,
+      "loss": 0.9209308624267578,
+      "mean_token_accuracy": 0.7674053549766541,
+      "num_tokens": 7082476.0,
+      "step": 1150
+    },
+    {
+      "entropy": 0.9472148299217225,
+      "epoch": 0.8608058608058609,
+      "grad_norm": 2.15625,
+      "learning_rate": 1.2717829903272889e-05,
+      "loss": 0.9266857147216797,
+      "mean_token_accuracy": 0.7626336240768432,
+      "num_tokens": 7239624.0,
+      "step": 1175
+    },
+    {
+      "entropy": 0.9480014276504517,
+      "epoch": 0.8791208791208791,
+      "grad_norm": 2.3125,
+      "learning_rate": 1.243124033115382e-05,
+      "loss": 0.9297947692871094,
+      "mean_token_accuracy": 0.762817633152008,
+      "num_tokens": 7394047.0,
+      "step": 1200
+    },
+    {
+      "epoch": 0.8791208791208791,
+      "eval_entropy": 0.9203641795449786,
+      "eval_loss": 0.8990768790245056,
+      "eval_mean_token_accuracy": 0.7723050812880198,
+      "eval_num_tokens": 7394047.0,
+      "eval_runtime": 8.072,
+      "eval_samples_per_second": 284.813,
+      "eval_steps_per_second": 17.84,
+      "step": 1200
+    },
+    {
+      "entropy": 0.9616302692890167,
+      "epoch": 0.8974358974358975,
+      "grad_norm": 2.640625,
+      "learning_rate": 1.2142512108353583e-05,
+      "loss": 0.9401410675048828,
+      "mean_token_accuracy": 0.7661498367786408,
+      "num_tokens": 7548323.0,
+      "step": 1225
+    },
+    {
+      "entropy": 0.9228850102424622,
+      "epoch": 0.9157509157509157,
+      "grad_norm": 2.21875,
+      "learning_rate": 1.1851899215855653e-05,
+      "loss": 0.8985552215576171,
+      "mean_token_accuracy": 0.7717998087406158,
+      "num_tokens": 7700481.0,
+      "step": 1250
+    },
+    {
+      "entropy": 0.8970145070552826,
+      "epoch": 0.9340659340659341,
+      "grad_norm": 2.296875,
+      "learning_rate": 1.1559657292501044e-05,
+      "loss": 0.87716064453125,
+      "mean_token_accuracy": 0.7763631820678711,
+      "num_tokens": 7852495.0,
+      "step": 1275
+    },
+    {
+      "entropy": 0.9301983499526978,
+      "epoch": 0.9523809523809523,
+      "grad_norm": 2.09375,
+      "learning_rate": 1.1266043410114557e-05,
+      "loss": 0.907692642211914,
+      "mean_token_accuracy": 0.7715549063682556,
+      "num_tokens": 8001324.0,
+      "step": 1300
+    },
+    {
+      "entropy": 0.91446182847023,
+      "epoch": 0.9706959706959707,
+      "grad_norm": 2.359375,
+      "learning_rate": 1.0971315847370463e-05,
+      "loss": 0.8939350891113281,
+      "mean_token_accuracy": 0.7720365560054779,
+      "num_tokens": 8161385.0,
+      "step": 1325
+    },
+    {
+      "entropy": 0.9196098625659943,
+      "epoch": 0.989010989010989,
+      "grad_norm": 2.109375,
+      "learning_rate": 1.06757338625966e-05,
+      "loss": 0.89285888671875,
+      "mean_token_accuracy": 0.7721686267852783,
+      "num_tokens": 8315619.0,
+      "step": 1350
+    },
+    {
+      "entropy": 0.9019503581523896,
+      "epoch": 1.0073260073260073,
+      "grad_norm": 1.8828125,
+      "learning_rate": 1.0379557465716696e-05,
+      "loss": 0.853975830078125,
+      "mean_token_accuracy": 0.7786379647254944,
+      "num_tokens": 8474116.0,
+      "step": 1375
+    },
+    {
+      "entropy": 0.8720257365703583,
+      "epoch": 1.0256410256410255,
+      "grad_norm": 2.140625,
+      "learning_rate": 1.0083047189531566e-05,
+      "loss": 0.8370319366455078,
+      "mean_token_accuracy": 0.7838277363777161,
+      "num_tokens": 8633284.0,
+      "step": 1400
+    },
+    {
+      "epoch": 1.0256410256410255,
+      "eval_entropy": 0.8584034711950355,
+      "eval_loss": 0.9003849029541016,
+      "eval_mean_token_accuracy": 0.7730862986710336,
+      "eval_num_tokens": 8633284.0,
+      "eval_runtime": 8.0586,
+      "eval_samples_per_second": 285.285,
+      "eval_steps_per_second": 17.869,
+      "step": 1400
+    },
+    {
+      "entropy": 0.8384023177623748,
+      "epoch": 1.043956043956044,
+      "grad_norm": 2.15625,
+      "learning_rate": 9.786463860540352e-06,
+      "loss": 0.7985468292236328,
+      "mean_token_accuracy": 0.7922709083557129,
+      "num_tokens": 8785152.0,
+      "step": 1425
+    },
+    {
+      "entropy": 0.848993946313858,
+      "epoch": 1.0622710622710623,
+      "grad_norm": 2.234375,
+      "learning_rate": 9.49006836950339e-06,
+      "loss": 0.8184638977050781,
+      "mean_token_accuracy": 0.7883424413204193,
+      "num_tokens": 8936429.0,
+      "step": 1450
+    },
+    {
+      "entropy": 0.8467873334884644,
+      "epoch": 1.0805860805860805,
+      "grad_norm": 2.140625,
+      "learning_rate": 9.194121441948596e-06,
+      "loss": 0.8056396484375,
+      "mean_token_accuracy": 0.7903980255126953,
+      "num_tokens": 9094633.0,
+      "step": 1475
+    },
+    {
+      "entropy": 0.8402684187889099,
+      "epoch": 1.098901098901099,
+      "grad_norm": 2.109375,
+      "learning_rate": 8.898883408823124e-06,
+      "loss": 0.8085630798339843,
+      "mean_token_accuracy": 0.7898635935783386,
+      "num_tokens": 9246323.0,
+      "step": 1500
+    },
+    {
+      "entropy": 0.8303942859172821,
+      "epoch": 1.1172161172161172,
+      "grad_norm": 2.0625,
+      "learning_rate": 8.604613977492212e-06,
+      "loss": 0.7932289123535157,
+      "mean_token_accuracy": 0.7939286470413208,
+      "num_tokens": 9395722.0,
+      "step": 1525
+    },
+    {
+      "entropy": 0.8481007432937622,
+      "epoch": 1.1355311355311355,
+      "grad_norm": 2.109375,
+      "learning_rate": 8.3115720032865e-06,
+      "loss": 0.8194800567626953,
+      "mean_token_accuracy": 0.7893430662155151,
+      "num_tokens": 9547484.0,
+      "step": 1550
+    },
+    {
+      "entropy": 0.8354680657386779,
+      "epoch": 1.1538461538461537,
+      "grad_norm": 2.265625,
+      "learning_rate": 8.02001526179883e-06,
+      "loss": 0.8005447387695312,
+      "mean_token_accuracy": 0.7913288366794586,
+      "num_tokens": 9699959.0,
+      "step": 1575
+    },
+    {
+      "entropy": 0.8411949121952057,
+      "epoch": 1.1721611721611722,
+      "grad_norm": 2.125,
+      "learning_rate": 7.730200222130932e-06,
+      "loss": 0.810956039428711,
+      "mean_token_accuracy": 0.7901583468914032,
+      "num_tokens": 9851149.0,
+      "step": 1600
+    },
+    {
+      "epoch": 1.1721611721611722,
+      "eval_entropy": 0.8544707364506192,
+      "eval_loss": 0.8998146057128906,
+      "eval_mean_token_accuracy": 0.7731748426126109,
+      "eval_num_tokens": 9851149.0,
+      "eval_runtime": 8.0715,
+      "eval_samples_per_second": 284.828,
+      "eval_steps_per_second": 17.84,
+      "step": 1600
+    },
+    {
+      "entropy": 0.8444626975059509,
+      "epoch": 1.1904761904761905,
+      "grad_norm": 2.328125,
+      "learning_rate": 7.442381821289248e-06,
+      "loss": 0.8108631896972657,
+      "mean_token_accuracy": 0.7912653696537018,
+      "num_tokens": 9998793.0,
+      "step": 1625
+    },
+    {
+      "entropy": 0.8223755669593811,
+      "epoch": 1.2087912087912087,
+      "grad_norm": 2.09375,
+      "learning_rate": 7.1568132399285794e-06,
+      "loss": 0.7869499969482422,
+      "mean_token_accuracy": 0.7936726534366607,
+      "num_tokens": 10152752.0,
+      "step": 1650
+    },
+    {
+      "entropy": 0.8130457592010498,
+      "epoch": 1.2271062271062272,
+      "grad_norm": 1.875,
+      "learning_rate": 6.8737456796406594e-06,
+      "loss": 0.78310546875,
+      "mean_token_accuracy": 0.7966435635089875,
+      "num_tokens": 10308001.0,
+      "step": 1675
+    },
+    {
+      "entropy": 0.8471330416202545,
+      "epoch": 1.2454212454212454,
+      "grad_norm": 2.046875,
+      "learning_rate": 6.593428141983595e-06,
+      "loss": 0.8101668548583985,
+      "mean_token_accuracy": 0.7890221488475799,
+      "num_tokens": 10459655.0,
+      "step": 1700
+    },
+    {
+      "entropy": 0.8247973656654358,
+      "epoch": 1.2637362637362637,
+      "grad_norm": 2.1875,
+      "learning_rate": 6.316107209446662e-06,
+      "loss": 0.796153564453125,
+      "mean_token_accuracy": 0.7926836657524109,
+      "num_tokens": 10615183.0,
+      "step": 1725
+    },
+    {
+      "entropy": 0.8476881444454193,
+      "epoch": 1.282051282051282,
+      "grad_norm": 2.1875,
+      "learning_rate": 6.042026828542942e-06,
+      "loss": 0.8173642730712891,
+      "mean_token_accuracy": 0.7872111082077027,
+      "num_tokens": 10774019.0,
+      "step": 1750
+    },
+    {
+      "entropy": 0.8313502633571624,
+      "epoch": 1.3003663003663004,
+      "grad_norm": 2.0,
+      "learning_rate": 5.7714280952207955e-06,
+      "loss": 0.8023670959472656,
+      "mean_token_accuracy": 0.7926879894733428,
+      "num_tokens": 10926453.0,
+      "step": 1775
+    },
+    {
+      "entropy": 0.8281714594364167,
+      "epoch": 1.3186813186813187,
+      "grad_norm": 2.296875,
+      "learning_rate": 5.504549042782779e-06,
+      "loss": 0.7919468688964844,
+      "mean_token_accuracy": 0.7943351435661316,
+      "num_tokens": 11076817.0,
+      "step": 1800
+    },
+    {
+      "epoch": 1.3186813186813187,
+      "eval_entropy": 0.8518242807024055,
+      "eval_loss": 0.8994762897491455,
+      "eval_mean_token_accuracy": 0.7733377065095637,
+      "eval_num_tokens": 11076817.0,
+      "eval_runtime": 8.0691,
+      "eval_samples_per_second": 284.912,
+      "eval_steps_per_second": 17.846,
+      "step": 1800
+    },
+    {
+      "entropy": 0.8253132402896881,
+      "epoch": 1.3369963369963371,
+      "grad_norm": 2.09375,
+      "learning_rate": 5.241624432498673e-06,
+      "loss": 0.7934477996826171,
+      "mean_token_accuracy": 0.7905769181251526,
+      "num_tokens": 11235359.0,
+      "step": 1825
+    },
+    {
+      "entropy": 0.8527077162265777,
+      "epoch": 1.3553113553113554,
+      "grad_norm": 2.25,
+      "learning_rate": 4.982885547096754e-06,
+      "loss": 0.8256228637695312,
+      "mean_token_accuracy": 0.7879673659801483,
+      "num_tokens": 11385249.0,
+      "step": 1850
+    },
+    {
+      "entropy": 0.8239582288265228,
+      "epoch": 1.3736263736263736,
+      "grad_norm": 2.09375,
+      "learning_rate": 4.728559987314975e-06,
+      "loss": 0.7923411560058594,
+      "mean_token_accuracy": 0.7940970385074615,
+      "num_tokens": 11539970.0,
+      "step": 1875
+    },
+    {
+      "entropy": 0.8423762857913971,
+      "epoch": 1.3919413919413919,
+      "grad_norm": 2.1875,
+      "learning_rate": 4.478871471691031e-06,
+      "loss": 0.8075601196289063,
+      "mean_token_accuracy": 0.7895544397830964,
+      "num_tokens": 11694684.0,
+      "step": 1900
+    },
+    {
+      "entropy": 0.8148546552658081,
+      "epoch": 1.4102564102564101,
+      "grad_norm": 2.125,
+      "learning_rate": 4.234039639767406e-06,
+      "loss": 0.7813822937011718,
+      "mean_token_accuracy": 0.7967073798179627,
+      "num_tokens": 11850074.0,
+      "step": 1925
+    },
+    {
+      "entropy": 0.840382571220398,
+      "epoch": 1.4285714285714286,
+      "grad_norm": 2.09375,
+      "learning_rate": 3.994279858884551e-06,
+      "loss": 0.8069898986816406,
+      "mean_token_accuracy": 0.7921997106075287,
+      "num_tokens": 12004729.0,
+      "step": 1950
+    },
+    {
+      "entropy": 0.8522861528396607,
+      "epoch": 1.4468864468864469,
+      "grad_norm": 2.03125,
+      "learning_rate": 3.7598030347321e-06,
+      "loss": 0.8208354187011718,
+      "mean_token_accuracy": 0.789222640991211,
+      "num_tokens": 12157974.0,
+      "step": 1975
+    },
+    {
+      "entropy": 0.8414350926876069,
+      "epoch": 1.4652014652014653,
+      "grad_norm": 2.078125,
+      "learning_rate": 3.530815425824797e-06,
+      "loss": 0.8093458557128906,
+      "mean_token_accuracy": 0.7892864966392517,
+      "num_tokens": 12306150.0,
+      "step": 2000
+    },
+    {
+      "epoch": 1.4652014652014653,
+      "eval_entropy": 0.8502832547658019,
+      "eval_loss": 0.8990073204040527,
+      "eval_mean_token_accuracy": 0.7732960196832815,
+      "eval_num_tokens": 12306150.0,
+      "eval_runtime": 8.068,
+      "eval_samples_per_second": 284.952,
+      "eval_steps_per_second": 17.848,
+      "step": 2000
+    },
+    {
+      "entropy": 0.8521664845943451,
+      "epoch": 1.4835164835164836,
+      "grad_norm": 2.0625,
+      "learning_rate": 3.307518462066344e-06,
+      "loss": 0.8166194915771484,
+      "mean_token_accuracy": 0.785648273229599,
+      "num_tokens": 12456962.0,
+      "step": 2025
+    },
+    {
+      "entropy": 0.8251003730297088,
+      "epoch": 1.5018315018315018,
+      "grad_norm": 2.203125,
+      "learning_rate": 3.0901085675607266e-06,
+      "loss": 0.7916029357910156,
+      "mean_token_accuracy": 0.7955028152465821,
+      "num_tokens": 12606544.0,
+      "step": 2050
+    },
+    {
+      "entropy": 0.8439912271499633,
+      "epoch": 1.52014652014652,
+      "grad_norm": 2.203125,
+      "learning_rate": 2.8787769878269667e-06,
+      "loss": 0.811083755493164,
+      "mean_token_accuracy": 0.7906605911254883,
+      "num_tokens": 12763473.0,
+      "step": 2075
+    },
+    {
+      "entropy": 0.8612873029708862,
+      "epoch": 1.5384615384615383,
+      "grad_norm": 2.21875,
+      "learning_rate": 2.673709621569167e-06,
+      "loss": 0.834039077758789,
+      "mean_token_accuracy": 0.786132972240448,
+      "num_tokens": 12917918.0,
+      "step": 2100
+    },
+    {
+      "entropy": 0.832386200428009,
+      "epoch": 1.5567765567765568,
+      "grad_norm": 2.15625,
+      "learning_rate": 2.475086857149982e-06,
+      "loss": 0.7950384521484375,
+      "mean_token_accuracy": 0.7901384067535401,
+      "num_tokens": 13072296.0,
+      "step": 2125
+    },
+    {
+      "entropy": 0.8401387500762939,
+      "epoch": 1.575091575091575,
+      "grad_norm": 2.140625,
+      "learning_rate": 2.2830834139112136e-06,
+      "loss": 0.8060906219482422,
+      "mean_token_accuracy": 0.7896384346485138,
+      "num_tokens": 13226077.0,
+      "step": 2150
+    },
+    {
+      "entropy": 0.8430456602573395,
+      "epoch": 1.5934065934065935,
+      "grad_norm": 2.28125,
+      "learning_rate": 2.097868188481217e-06,
+      "loss": 0.8120394134521485,
+      "mean_token_accuracy": 0.7886982035636901,
+      "num_tokens": 13379631.0,
+      "step": 2175
+    },
+    {
+      "entropy": 0.8341074979305267,
+      "epoch": 1.6117216117216118,
+      "grad_norm": 2.28125,
+      "learning_rate": 1.919604106204298e-06,
+      "loss": 0.7923731231689453,
+      "mean_token_accuracy": 0.7932259738445282,
+      "num_tokens": 13537431.0,
+      "step": 2200
+    },
+    {
+      "epoch": 1.6117216117216118,
+      "eval_entropy": 0.8466265425086021,
+      "eval_loss": 0.8991615176200867,
+      "eval_mean_token_accuracy": 0.7732404433190823,
+      "eval_num_tokens": 13537431.0,
+      "eval_runtime": 8.0784,
+      "eval_samples_per_second": 284.585,
+      "eval_steps_per_second": 17.825,
+      "step": 2200
+    },
+    {
+      "entropy": 0.829889565706253,
+      "epoch": 1.63003663003663,
+      "grad_norm": 2.09375,
+      "learning_rate": 1.7484479778227281e-06,
+      "loss": 0.7963677215576171,
+      "mean_token_accuracy": 0.7930803084373474,
+      "num_tokens": 13696124.0,
+      "step": 2225
+    },
+    {
+      "entropy": 0.8456670987606049,
+      "epoch": 1.6483516483516483,
+      "grad_norm": 2.125,
+      "learning_rate": 1.5845503615375534e-06,
+      "loss": 0.8171965789794922,
+      "mean_token_accuracy": 0.78998281955719,
+      "num_tokens": 13850353.0,
+      "step": 2250
+    },
+    {
+      "entropy": 0.834287303686142,
+      "epoch": 1.6666666666666665,
+      "grad_norm": 2.203125,
+      "learning_rate": 1.4280554305694205e-06,
+      "loss": 0.8009786987304688,
+      "mean_token_accuracy": 0.7929751622676849,
+      "num_tokens": 14002830.0,
+      "step": 2275
+    },
+    {
+      "entropy": 0.8210822236537934,
+      "epoch": 1.684981684981685,
+      "grad_norm": 2.171875,
+      "learning_rate": 1.2791008463360077e-06,
+      "loss": 0.7856560516357421,
+      "mean_token_accuracy": 0.7937294292449951,
+      "num_tokens": 14159802.0,
+      "step": 2300
+    },
+    {
+      "entropy": 0.8226444447040557,
+      "epoch": 1.7032967032967035,
+      "grad_norm": 2.09375,
+      "learning_rate": 1.1378176373575977e-06,
+      "loss": 0.7973332977294922,
+      "mean_token_accuracy": 0.7935956382751465,
+      "num_tokens": 14315490.0,
+      "step": 2325
+    },
+    {
+      "entropy": 0.8418577921390533,
+      "epoch": 1.7216117216117217,
+      "grad_norm": 2.203125,
+      "learning_rate": 1.004330083997278e-06,
+      "loss": 0.8100568389892578,
+      "mean_token_accuracy": 0.79057302236557,
+      "num_tokens": 14471457.0,
+      "step": 2350
+    },
+    {
+      "entropy": 0.8569370365142822,
+      "epoch": 1.73992673992674,
+      "grad_norm": 2.296875,
+      "learning_rate": 8.787556091372207e-07,
+      "loss": 0.8271647644042969,
+      "mean_token_accuracy": 0.783940087556839,
+      "num_tokens": 14627084.0,
+      "step": 2375
+    },
+    {
+      "entropy": 0.8276915633678437,
+      "epoch": 1.7582417582417582,
+      "grad_norm": 2.171875,
+      "learning_rate": 7.612046748871327e-07,
+      "loss": 0.8014369201660156,
+      "mean_token_accuracy": 0.790752956867218,
+      "num_tokens": 14772996.0,
+      "step": 2400
+    },
+    {
+      "epoch": 1.7582417582417582,
+      "eval_entropy": 0.8465850080053011,
+      "eval_loss": 0.899135172367096,
+      "eval_mean_token_accuracy": 0.7732518298758401,
+      "eval_num_tokens": 14772996.0,
+      "eval_runtime": 8.0715,
+      "eval_samples_per_second": 284.83,
+      "eval_steps_per_second": 17.841,
+      "step": 2400
+    },
+    {
+      "entropy": 0.8189385521411896,
+      "epoch": 1.7765567765567765,
+      "grad_norm": 2.09375,
+      "learning_rate": 6.517806854158204e-07,
+      "loss": 0.7890790557861328,
+      "mean_token_accuracy": 0.7950505125522613,
+      "num_tokens": 14926511.0,
+      "step": 2425
+    },
+    {
+      "entropy": 0.8378893113136292,
+      "epoch": 1.7948717948717947,
+      "grad_norm": 2.359375,
+      "learning_rate": 5.505798959912723e-07,
+      "loss": 0.8117930603027343,
+      "mean_token_accuracy": 0.7887657046318054,
+      "num_tokens": 15081687.0,
+      "step": 2450
+    },
+    {
+      "entropy": 0.8320703601837158,
+      "epoch": 1.8131868131868132,
+      "grad_norm": 2.140625,
+      "learning_rate": 4.576913283093098e-07,
+      "loss": 0.8038975524902344,
+      "mean_token_accuracy": 0.7904533529281617,
+      "num_tokens": 15236278.0,
+      "step": 2475
+    },
+    {
+      "entropy": 0.8401167035102844,
+      "epoch": 1.8315018315018317,
+      "grad_norm": 2.234375,
+      "learning_rate": 3.7319669218529475e-07,
+      "loss": 0.8033819580078125,
+      "mean_token_accuracy": 0.7910889124870301,
+      "num_tokens": 15391169.0,
+      "step": 2500
+    },
+    {
+      "entropy": 0.858134560585022,
+      "epoch": 1.84981684981685,
+      "grad_norm": 2.359375,
+      "learning_rate": 2.971703136777315e-07,
+      "loss": 0.8318187713623046,
+      "mean_token_accuracy": 0.784791008234024,
+      "num_tokens": 15548066.0,
+      "step": 2525
+    },
+    {
+      "entropy": 0.8192259454727173,
+      "epoch": 1.8681318681318682,
+      "grad_norm": 1.9453125,
+      "learning_rate": 2.2967906970705722e-07,
+      "loss": 0.7859656524658203,
+      "mean_token_accuracy": 0.7967568933963776,
+      "num_tokens": 15699746.0,
+      "step": 2550
+    },
+    {
+      "entropy": 0.8540216779708862,
+      "epoch": 1.8864468864468864,
+      "grad_norm": 2.171875,
+      "learning_rate": 1.7078232922706495e-07,
+      "loss": 0.8298044586181641,
+      "mean_token_accuracy": 0.7818279254436493,
+      "num_tokens": 15856690.0,
+      "step": 2575
+    },
+    {
+      "entropy": 0.8150307929515839,
+      "epoch": 1.9047619047619047,
+      "grad_norm": 2.109375,
+      "learning_rate": 1.2053190100077106e-07,
+      "loss": 0.7887921905517579,
+      "mean_token_accuracy": 0.7932768344879151,
+      "num_tokens": 16013738.0,
+      "step": 2600
+    },
+    {
+      "epoch": 1.9047619047619047,
+      "eval_entropy": 0.8467293654878935,
+      "eval_loss": 0.8991142511367798,
+      "eval_mean_token_accuracy": 0.7734044496383932,
+      "eval_num_tokens": 16013738.0,
+      "eval_runtime": 8.0684,
+      "eval_samples_per_second": 284.939,
+      "eval_steps_per_second": 17.847,
+      "step": 2600
+    }
+  ],
+  "logging_steps": 25,
+  "max_steps": 2730,
+  "num_input_tokens_seen": 0,
+  "num_train_epochs": 2,
+  "save_steps": 200,
+  "stateful_callbacks": {
+    "TrainerControl": {
+      "args": {
+        "should_epoch_stop": false,
+        "should_evaluate": false,
+        "should_log": false,
+        "should_save": true,
+        "should_training_stop": false
+      },
+      "attributes": {}
+    }
+  },
+  "total_flos": 6.478793611126733e+16,
+  "train_batch_size": 16,
+  "trial_name": null,
+  "trial_params": null
+}

checkpoint-2600/training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1f3eec82a82cc2cb4f2c124367cd9fb480a301c5cdfda379c7339e6e590e5bd3
+size 5585

checkpoint-2730/chat_template.jinja

@@ -0,0 +1,64 @@
+{{- bos_token -}}
+{%- set keep_past_thinking = keep_past_thinking | default(false) -%}
+{%- set ns = namespace(system_prompt="") -%}
+{%- if messages[0]["role"] == "system" -%}
+    {%- set sys_content = messages[0]["content"] -%}
+    {%- if sys_content is not string -%}
+        {%- for item in sys_content -%}
+            {%- if item["type"] == "text" -%}
+                {%- set ns.system_prompt = ns.system_prompt + item["text"] -%}
+            {%- endif -%}
+        {%- endfor -%}
+    {%- else -%}
+        {%- set ns.system_prompt = sys_content -%}
+    {%- endif -%}
+    {%- set messages = messages[1:] -%}
+{%- endif -%}
+{%- if tools -%}
+    {%- set ns.system_prompt = ns.system_prompt + ("\n" if ns.system_prompt else "") + "List of tools: [" -%}
+    {%- for tool in tools -%}
+        {%- if tool is not string -%}
+            {%- set tool = tool | tojson -%}
+        {%- endif -%}
+        {%- set ns.system_prompt = ns.system_prompt + tool -%}
+        {%- if not loop.last -%}
+            {%- set ns.system_prompt = ns.system_prompt + ", " -%}
+        {%- endif -%}
+    {%- endfor -%}
+    {%- set ns.system_prompt = ns.system_prompt + "]" -%}
+{%- endif -%}
+{%- if ns.system_prompt -%}
+    {{- "<|im_start|>system\n" + ns.system_prompt + "<|im_end|>\n" -}}
+{%- endif -%}
+{%- set ns.last_assistant_index = -1 -%}
+{%- for message in messages -%}
+    {%- if message["role"] == "assistant" -%}
+        {%- set ns.last_assistant_index = loop.index0 -%}
+    {%- endif -%}
+{%- endfor -%}
+{%- for message in messages -%}
+    {{- "<|im_start|>" + message["role"] + "\n" -}}
+    {%- set content = message["content"] -%}
+    {%- if content is not string -%}
+        {%- set ns.content = "" -%}
+        {%- for item in content -%}
+            {%- if item["type"] == "image" -%}
+                {%- set ns.content = ns.content + "<image>" -%}
+            {%- elif item["type"] == "text" -%}
+                {%- set ns.content = ns.content + item["text"] -%}
+            {%- else -%}
+                {%- set ns.content = ns.content + item | tojson -%}
+            {%- endif -%}
+        {%- endfor -%}
+        {%- set content = ns.content -%}
+    {%- endif -%}
+    {%- if message["role"] == "assistant" and not keep_past_thinking and loop.index0 != ns.last_assistant_index -%}
+        {%- if "</think>" in content -%}
+            {%- set content = content.split("</think>")[-1] | trim -%}
+        {%- endif -%}
+    {%- endif -%}
+    {{- content + "<|im_end|>\n" -}}
+{%- endfor -%}
+{%- if add_generation_prompt -%}
+    {{- "<|im_start|>assistant\n" -}}
+{%- endif -%}

checkpoint-2730/config.json

@@ -0,0 +1,60 @@
+{
+  "architectures": [
+    "Lfm2ForCausalLM"
+  ],
+  "block_auto_adjust_ff_dim": true,
+  "block_dim": 1024,
+  "block_ffn_dim_multiplier": 1.0,
+  "block_mlp_init_scale": 1.0,
+  "block_multiple_of": 256,
+  "block_norm_eps": 1e-05,
+  "block_out_init_scale": 1.0,
+  "block_use_swiglu": true,
+  "block_use_xavier_init": true,
+  "bos_token_id": 1,
+  "conv_L_cache": 3,
+  "conv_bias": false,
+  "conv_dim": 1024,
+  "conv_use_xavier_init": true,
+  "dtype": "bfloat16",
+  "eos_token_id": 7,
+  "full_attn_idxs": null,
+  "hidden_size": 1024,
+  "initializer_range": 0.02,
+  "intermediate_size": 6656,
+  "layer_types": [
+    "conv",
+    "conv",
+    "full_attention",
+    "conv",
+    "conv",
+    "full_attention",
+    "conv",
+    "conv",
+    "full_attention",
+    "conv",
+    "full_attention",
+    "conv",
+    "full_attention",
+    "conv",
+    "full_attention",
+    "conv"
+  ],
+  "max_position_embeddings": 128000,
+  "model_type": "lfm2",
+  "norm_eps": 1e-05,
+  "num_attention_heads": 16,
+  "num_heads": 16,
+  "num_hidden_layers": 16,
+  "num_key_value_heads": 8,
+  "pad_token_id": 0,
+  "rope_parameters": {
+    "rope_theta": 1000000.0,
+    "rope_type": "default"
+  },
+  "tie_word_embeddings": true,
+  "transformers_version": "5.5.0",
+  "use_cache": false,
+  "use_pos_enc": true,
+  "vocab_size": 65536
+}

checkpoint-2730/generation_config.json

@@ -0,0 +1,9 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 1,
+  "eos_token_id": [
+    7
+  ],
+  "pad_token_id": 0,
+  "transformers_version": "5.5.0"
+}

checkpoint-2730/model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b457272936237b41f0e45ff4f4c6cbbbf39341c6779ca18c9e2a75766555ee3b
+size 708984464

checkpoint-2730/optimizer.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c344d991f5f277dbd4ea1e3501c5f329f9af6d61c9100451fae42daaf16f7180
+size 1418061003

checkpoint-2730/rng_state.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f6657ccd5ba73eb2588fe6c69638f02621253e47f1271867fd3af0b8ff5c9b2a
+size 14645

checkpoint-2730/scheduler.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:39f96fd5fdf394c9c5883cb0134dd09bc95d65d439f9fbd368008433206ed4fb
+size 1465

checkpoint-2730/tokenizer_config.json

@@ -0,0 +1,21 @@
+{
+  "backend": "tokenizers",
+  "bos_token": "<|startoftext|>",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|im_end|>",
+  "extra_special_tokens": [],
+  "is_local": false,
+  "legacy": false,
+  "model_input_names": [
+    "input_ids",
+    "attention_mask"
+  ],
+  "model_max_length": 1000000000000000019884624838656,
+  "model_specific_special_tokens": {},
+  "pad_token": "<|pad|>",
+  "sp_model_kwargs": {},
+  "spaces_between_special_tokens": false,
+  "tokenizer_class": "TokenizersBackend",
+  "use_default_system_prompt": false,
+  "use_fast": true
+}

checkpoint-2730/trainer_state.json

@@ -0,0 +1,1278 @@
+{
+  "best_global_step": 2000,
+  "best_metric": 0.8990073204040527,
+  "best_model_checkpoint": "lfm2.5-350m-balanced/checkpoint-2000",
+  "epoch": 2.0,
+  "eval_steps": 200,
+  "global_step": 2730,
+  "is_hyper_param_search": false,
+  "is_local_process_zero": true,
+  "is_world_process_zero": true,
+  "log_history": [
+    {
+      "entropy": 1.20119291305542,
+      "epoch": 0.018315018315018316,
+      "grad_norm": 10.4375,
+      "learning_rate": 5.853658536585366e-06,
+      "loss": 2.8065673828125,
+      "mean_token_accuracy": 0.57585333943367,
+      "num_tokens": 148430.0,
+      "step": 25
+    },
+    {
+      "entropy": 1.278236083984375,
+      "epoch": 0.03663003663003663,
+      "grad_norm": 4.125,
+      "learning_rate": 1.1951219512195123e-05,
+      "loss": 1.506644287109375,
+      "mean_token_accuracy": 0.7009313237667084,
+      "num_tokens": 301448.0,
+      "step": 50
+    },
+    {
+      "entropy": 1.0903868675231934,
+      "epoch": 0.054945054945054944,
+      "grad_norm": 2.609375,
+      "learning_rate": 1.804878048780488e-05,
+      "loss": 1.0758729553222657,
+      "mean_token_accuracy": 0.7404501688480377,
+      "num_tokens": 458432.0,
+      "step": 75
+    },
+    {
+      "entropy": 1.0081040942668915,
+      "epoch": 0.07326007326007326,
+      "grad_norm": 2.4375,
+      "learning_rate": 1.9997966159416884e-05,
+      "loss": 0.9917147827148437,
+      "mean_token_accuracy": 0.7522138917446136,
+      "num_tokens": 616048.0,
+      "step": 100
+    },
+    {
+      "entropy": 0.9854830229282379,
+      "epoch": 0.09157509157509157,
+      "grad_norm": 2.46875,
+      "learning_rate": 1.99875879788391e-05,
+      "loss": 0.9723490905761719,
+      "mean_token_accuracy": 0.7565386164188385,
+      "num_tokens": 772466.0,
+      "step": 125
+    },
+    {
+      "entropy": 0.9745642852783203,
+      "epoch": 0.10989010989010989,
+      "grad_norm": 2.25,
+      "learning_rate": 1.996842417490666e-05,
+      "loss": 0.9625119018554688,
+      "mean_token_accuracy": 0.760125788450241,
+      "num_tokens": 930723.0,
+      "step": 150
+    },
+    {
+      "entropy": 0.9700245261192322,
+      "epoch": 0.1282051282051282,
+      "grad_norm": 2.234375,
+      "learning_rate": 1.99404916051395e-05,
+      "loss": 0.9550397491455078,
+      "mean_token_accuracy": 0.7578657758235932,
+      "num_tokens": 1091160.0,
+      "step": 175
+    },
+    {
+      "entropy": 0.9669338822364807,
+      "epoch": 0.14652014652014653,
+      "grad_norm": 2.359375,
+      "learning_rate": 1.9903814840538914e-05,
+      "loss": 0.9478360748291016,
+      "mean_token_accuracy": 0.7623070132732391,
+      "num_tokens": 1245781.0,
+      "step": 200
+    },
+    {
+      "epoch": 0.14652014652014653,
+      "eval_entropy": 0.9581337045464251,
+      "eval_loss": 0.9430390000343323,
+      "eval_mean_token_accuracy": 0.7639545847972234,
+      "eval_num_tokens": 1245781.0,
+      "eval_runtime": 8.0902,
+      "eval_samples_per_second": 284.171,
+      "eval_steps_per_second": 17.799,
+      "step": 200
+    },
+    {
+      "entropy": 0.9653169775009155,
+      "epoch": 0.16483516483516483,
+      "grad_norm": 2.640625,
+      "learning_rate": 1.9858426143973623e-05,
+      "loss": 0.9534239959716797,
+      "mean_token_accuracy": 0.7631052339076996,
+      "num_tokens": 1395650.0,
+      "step": 225
+    },
+    {
+      "entropy": 0.962496577501297,
+      "epoch": 0.18315018315018314,
+      "grad_norm": 2.234375,
+      "learning_rate": 1.9804365441799555e-05,
+      "loss": 0.9441399383544922,
+      "mean_token_accuracy": 0.7629920542240143,
+      "num_tokens": 1553420.0,
+      "step": 250
+    },
+    {
+      "entropy": 0.9607322406768799,
+      "epoch": 0.20146520146520147,
+      "grad_norm": 2.46875,
+      "learning_rate": 1.9741680288738495e-05,
+      "loss": 0.939505615234375,
+      "mean_token_accuracy": 0.7633682239055634,
+      "num_tokens": 1708083.0,
+      "step": 275
+    },
+    {
+      "entropy": 0.9339856076240539,
+      "epoch": 0.21978021978021978,
+      "grad_norm": 2.203125,
+      "learning_rate": 1.967042582604635e-05,
+      "loss": 0.9153070831298828,
+      "mean_token_accuracy": 0.767605767250061,
+      "num_tokens": 1862545.0,
+      "step": 300
+    },
+    {
+      "entropy": 0.9721270000934601,
+      "epoch": 0.23809523809523808,
+      "grad_norm": 2.265625,
+      "learning_rate": 1.9590664733007947e-05,
+      "loss": 0.959111328125,
+      "mean_token_accuracy": 0.7571033167839051,
+      "num_tokens": 2017595.0,
+      "step": 325
+    },
+    {
+      "entropy": 0.961563161611557,
+      "epoch": 0.2564102564102564,
+      "grad_norm": 2.3125,
+      "learning_rate": 1.9502467171800902e-05,
+      "loss": 0.9458185577392578,
+      "mean_token_accuracy": 0.7628096437454224,
+      "num_tokens": 2166320.0,
+      "step": 350
+    },
+    {
+      "entropy": 0.9491504514217377,
+      "epoch": 0.27472527472527475,
+      "grad_norm": 2.34375,
+      "learning_rate": 1.940591072577719e-05,
+      "loss": 0.9269615173339844,
+      "mean_token_accuracy": 0.7635479891300201,
+      "num_tokens": 2325678.0,
+      "step": 375
+    },
+    {
+      "entropy": 0.9450718343257904,
+      "epoch": 0.29304029304029305,
+      "grad_norm": 2.28125,
+      "learning_rate": 1.930108033121663e-05,
+      "loss": 0.9313880920410156,
+      "mean_token_accuracy": 0.7636011290550232,
+      "num_tokens": 2481053.0,
+      "step": 400
+    },
+    {
+      "epoch": 0.29304029304029305,
+      "eval_entropy": 0.9506676958666908,
+      "eval_loss": 0.9265689849853516,
+      "eval_mean_token_accuracy": 0.7673345183332761,
+      "eval_num_tokens": 2481053.0,
+      "eval_runtime": 8.0719,
+      "eval_samples_per_second": 284.816,
+      "eval_steps_per_second": 17.84,
+      "step": 400
+    },
+    {
+      "entropy": 0.9616206979751587,
+      "epoch": 0.31135531135531136,
+      "grad_norm": 2.28125,
+      "learning_rate": 1.9188068202612317e-05,
+      "loss": 0.9455902862548828,
+      "mean_token_accuracy": 0.7630793690681458,
+      "num_tokens": 2633845.0,
+      "step": 425
+    },
+    {
+      "entropy": 0.9392594003677368,
+      "epoch": 0.32967032967032966,
+      "grad_norm": 2.3125,
+      "learning_rate": 1.906697375155377e-05,
+      "loss": 0.9156442260742188,
+      "mean_token_accuracy": 0.7659864926338196,
+      "num_tokens": 2790745.0,
+      "step": 450
+    },
+    {
+      "entropy": 0.9245667338371277,
+      "epoch": 0.34798534798534797,
+      "grad_norm": 2.203125,
+      "learning_rate": 1.8937903499279104e-05,
+      "loss": 0.903341064453125,
+      "mean_token_accuracy": 0.7713142216205597,
+      "num_tokens": 2943444.0,
+      "step": 475
+    },
+    {
+      "entropy": 0.912586669921875,
+      "epoch": 0.3663003663003663,
+      "grad_norm": 2.390625,
+      "learning_rate": 1.880097098297319e-05,
+      "loss": 0.8922891235351562,
+      "mean_token_accuracy": 0.771296215057373,
+      "num_tokens": 3098959.0,
+      "step": 500
+    },
+    {
+      "entropy": 0.9415633583068848,
+      "epoch": 0.38461538461538464,
+      "grad_norm": 2.15625,
+      "learning_rate": 1.8656296655894205e-05,
+      "loss": 0.9213317108154296,
+      "mean_token_accuracy": 0.7668669641017913,
+      "num_tokens": 3248382.0,
+      "step": 525
+    },
+    {
+      "entropy": 0.9617620420455932,
+      "epoch": 0.40293040293040294,
+      "grad_norm": 2.109375,
+      "learning_rate": 1.8504007781416385e-05,
+      "loss": 0.9434718322753907,
+      "mean_token_accuracy": 0.7617218482494355,
+      "num_tokens": 3409803.0,
+      "step": 550
+    },
+    {
+      "entropy": 0.9193656361103058,
+      "epoch": 0.42124542124542125,
+      "grad_norm": 2.0625,
+      "learning_rate": 1.8344238321082316e-05,
+      "loss": 0.901815185546875,
+      "mean_token_accuracy": 0.7707051420211792,
+      "num_tokens": 3559416.0,
+      "step": 575
+    },
+    {
+      "entropy": 0.9518638348579407,
+      "epoch": 0.43956043956043955,
+      "grad_norm": 2.15625,
+      "learning_rate": 1.817712881676307e-05,
+      "loss": 0.9327508544921875,
+      "mean_token_accuracy": 0.7644142961502075,
+      "num_tokens": 3710110.0,
+      "step": 600
+    },
+    {
+      "epoch": 0.43956043956043955,
+      "eval_entropy": 0.9517907314002514,
+      "eval_loss": 0.9172877073287964,
+      "eval_mean_token_accuracy": 0.7684686299827364,
+      "eval_num_tokens": 3710110.0,
+      "eval_runtime": 8.0887,
+      "eval_samples_per_second": 284.222,
+      "eval_steps_per_second": 17.803,
+      "step": 600
+    },
+    {
+      "entropy": 0.9130004584789276,
+      "epoch": 0.45787545787545786,
+      "grad_norm": 2.21875,
+      "learning_rate": 1.8002826267029977e-05,
+      "loss": 0.8941175079345703,
+      "mean_token_accuracy": 0.7705032932758331,
+      "num_tokens": 3862020.0,
+      "step": 625
+    },
+    {
+      "entropy": 0.9252553701400756,
+      "epoch": 0.47619047619047616,
+      "grad_norm": 2.15625,
+      "learning_rate": 1.7821483997846722e-05,
+      "loss": 0.9165068054199219,
+      "mean_token_accuracy": 0.7672811985015869,
+      "num_tokens": 4018231.0,
+      "step": 650
+    },
+    {
+      "entropy": 0.9462221479415893,
+      "epoch": 0.4945054945054945,
+      "grad_norm": 2.125,
+      "learning_rate": 1.763326152769551e-05,
+      "loss": 0.9259709167480469,
+      "mean_token_accuracy": 0.7658140981197357,
+      "num_tokens": 4173058.0,
+      "step": 675
+    },
+    {
+      "entropy": 0.9479129528999328,
+      "epoch": 0.5128205128205128,
+      "grad_norm": 2.234375,
+      "learning_rate": 1.743832442725599e-05,
+      "loss": 0.9296504974365234,
+      "mean_token_accuracy": 0.7648407304286957,
+      "num_tokens": 4327983.0,
+      "step": 700
+    },
+    {
+      "entropy": 0.9435758924484253,
+      "epoch": 0.5311355311355311,
+      "grad_norm": 2.046875,
+      "learning_rate": 1.7236844173760286e-05,
+      "loss": 0.9163207244873047,
+      "mean_token_accuracy": 0.7653989827632904,
+      "num_tokens": 4482610.0,
+      "step": 725
+    },
+    {
+      "entropy": 0.9563624393939972,
+      "epoch": 0.5494505494505495,
+      "grad_norm": 2.296875,
+      "learning_rate": 1.7028998000152368e-05,
+      "loss": 0.942515869140625,
+      "mean_token_accuracy": 0.7614016819000244,
+      "num_tokens": 4638096.0,
+      "step": 750
+    },
+    {
+      "entropy": 0.9195479357242584,
+      "epoch": 0.5677655677655677,
+      "grad_norm": 2.3125,
+      "learning_rate": 1.681496873918434e-05,
+      "loss": 0.8994998931884766,
+      "mean_token_accuracy": 0.7696708023548127,
+      "num_tokens": 4796965.0,
+      "step": 775
+    },
+    {
+      "entropy": 0.9382291078567505,
+      "epoch": 0.5860805860805861,
+      "grad_norm": 2.453125,
+      "learning_rate": 1.659494466258685e-05,
+      "loss": 0.9111621856689454,
+      "mean_token_accuracy": 0.7701178181171418,
+      "num_tokens": 4954572.0,
+      "step": 800
+    },
+    {
+      "epoch": 0.5860805860805861,
+      "eval_entropy": 0.9060760368075635,
+      "eval_loss": 0.9103703498840332,
+      "eval_mean_token_accuracy": 0.7709767441782687,
+      "eval_num_tokens": 4954572.0,
+      "eval_runtime": 8.0596,
+      "eval_samples_per_second": 285.248,
+      "eval_steps_per_second": 17.867,
+      "step": 800
+    },
+    {
+      "entropy": 0.9392394709587097,
+      "epoch": 0.6043956043956044,
+      "grad_norm": 2.34375,
+      "learning_rate": 1.6369119315455067e-05,
+      "loss": 0.9231902313232422,
+      "mean_token_accuracy": 0.7648055994510651,
+      "num_tokens": 5107234.0,
+      "step": 825
+    },
+    {
+      "entropy": 0.9370434725284577,
+      "epoch": 0.6227106227106227,
+      "grad_norm": 2.0625,
+      "learning_rate": 1.613769134599595e-05,
+      "loss": 0.9138022613525391,
+      "mean_token_accuracy": 0.7679059386253357,
+      "num_tokens": 5256697.0,
+      "step": 850
+    },
+    {
+      "entropy": 0.9104315054416656,
+      "epoch": 0.6410256410256411,
+      "grad_norm": 2.15625,
+      "learning_rate": 1.590086433078652e-05,
+      "loss": 0.8872809600830078,
+      "mean_token_accuracy": 0.7746103525161743,
+      "num_tokens": 5406816.0,
+      "step": 875
+    },
+    {
+      "entropy": 0.9144675505161285,
+      "epoch": 0.6593406593406593,
+      "grad_norm": 2.125,
+      "learning_rate": 1.5658846595696868e-05,
+      "loss": 0.8894432830810547,
+      "mean_token_accuracy": 0.7708098363876342,
+      "num_tokens": 5558294.0,
+      "step": 900
+    },
+    {
+      "entropy": 0.9206836307048798,
+      "epoch": 0.6776556776556777,
+      "grad_norm": 2.125,
+      "learning_rate": 1.541185103263545e-05,
+      "loss": 0.8961500549316406,
+      "mean_token_accuracy": 0.771810313463211,
+      "num_tokens": 5712111.0,
+      "step": 925
+    },
+    {
+      "entropy": 0.9454787099361419,
+      "epoch": 0.6959706959706959,
+      "grad_norm": 2.015625,
+      "learning_rate": 1.5160094912277816e-05,
+      "loss": 0.9270650482177735,
+      "mean_token_accuracy": 0.7659087121486664,
+      "num_tokens": 5867502.0,
+      "step": 950
+    },
+    {
+      "entropy": 0.9311832976341248,
+      "epoch": 0.7142857142857143,
+      "grad_norm": 2.3125,
+      "learning_rate": 1.4903799692943575e-05,
+      "loss": 0.9082266235351563,
+      "mean_token_accuracy": 0.7696301078796387,
+      "num_tokens": 6017997.0,
+      "step": 975
+    },
+    {
+      "entropy": 0.9084031569957733,
+      "epoch": 0.7326007326007326,
+      "grad_norm": 2.265625,
+      "learning_rate": 1.4643190825789646e-05,
+      "loss": 0.8856626129150391,
+      "mean_token_accuracy": 0.7754030025005341,
+      "num_tokens": 6170646.0,
+      "step": 1000
+    },
+    {
+      "epoch": 0.7326007326007326,
+      "eval_entropy": 0.9303645570245054,
+      "eval_loss": 0.9040034413337708,
+      "eval_mean_token_accuracy": 0.7716102877424823,
+      "eval_num_tokens": 6170646.0,
+      "eval_runtime": 8.0677,
+      "eval_samples_per_second": 284.963,
+      "eval_steps_per_second": 17.849,
+      "step": 1000
+    },
+    {
+      "entropy": 0.9288157105445862,
+      "epoch": 0.7509157509157509,
+      "grad_norm": 2.125,
+      "learning_rate": 1.4378497556491212e-05,
+      "loss": 0.9045313262939453,
+      "mean_token_accuracy": 0.7689747631549835,
+      "num_tokens": 6325950.0,
+      "step": 1025
+    },
+    {
+      "entropy": 0.9248845767974854,
+      "epoch": 0.7692307692307693,
+      "grad_norm": 2.296875,
+      "learning_rate": 1.410995272358482e-05,
+      "loss": 0.9021368408203125,
+      "mean_token_accuracy": 0.7702593433856965,
+      "num_tokens": 6475837.0,
+      "step": 1050
+    },
+    {
+      "entropy": 0.9433277988433838,
+      "epoch": 0.7875457875457875,
+      "grad_norm": 2.03125,
+      "learning_rate": 1.383779255365097e-05,
+      "loss": 0.9240187072753906,
+      "mean_token_accuracy": 0.7659425389766693,
+      "num_tokens": 6625842.0,
+      "step": 1075
+    },
+    {
+      "entropy": 0.9574247586727143,
+      "epoch": 0.8058608058608059,
+      "grad_norm": 2.265625,
+      "learning_rate": 1.356225645351645e-05,
+      "loss": 0.9296712493896484,
+      "mean_token_accuracy": 0.7633269345760345,
+      "num_tokens": 6777307.0,
+      "step": 1100
+    },
+    {
+      "entropy": 0.9137916600704193,
+      "epoch": 0.8241758241758241,
+      "grad_norm": 2.234375,
+      "learning_rate": 1.3283586799659083e-05,
+      "loss": 0.8911846160888672,
+      "mean_token_accuracy": 0.7715334737300873,
+      "num_tokens": 6932314.0,
+      "step": 1125
+    },
+    {
+      "entropy": 0.9397987961769104,
+      "epoch": 0.8424908424908425,
+      "grad_norm": 2.171875,
+      "learning_rate": 1.3002028725000296e-05,
+      "loss": 0.9209308624267578,
+      "mean_token_accuracy": 0.7674053549766541,
+      "num_tokens": 7082476.0,
+      "step": 1150
+    },
+    {
+      "entropy": 0.9472148299217225,
+      "epoch": 0.8608058608058609,
+      "grad_norm": 2.15625,
+      "learning_rate": 1.2717829903272889e-05,
+      "loss": 0.9266857147216797,
+      "mean_token_accuracy": 0.7626336240768432,
+      "num_tokens": 7239624.0,
+      "step": 1175
+    },
+    {
+      "entropy": 0.9480014276504517,
+      "epoch": 0.8791208791208791,
+      "grad_norm": 2.3125,
+      "learning_rate": 1.243124033115382e-05,
+      "loss": 0.9297947692871094,
+      "mean_token_accuracy": 0.762817633152008,
+      "num_tokens": 7394047.0,
+      "step": 1200
+    },
+    {
+      "epoch": 0.8791208791208791,
+      "eval_entropy": 0.9203641795449786,
+      "eval_loss": 0.8990768790245056,
+      "eval_mean_token_accuracy": 0.7723050812880198,
+      "eval_num_tokens": 7394047.0,
+      "eval_runtime": 8.072,
+      "eval_samples_per_second": 284.813,
+      "eval_steps_per_second": 17.84,
+      "step": 1200
+    },
+    {
+      "entropy": 0.9616302692890167,
+      "epoch": 0.8974358974358975,
+      "grad_norm": 2.640625,
+      "learning_rate": 1.2142512108353583e-05,
+      "loss": 0.9401410675048828,
+      "mean_token_accuracy": 0.7661498367786408,
+      "num_tokens": 7548323.0,
+      "step": 1225
+    },
+    {
+      "entropy": 0.9228850102424622,
+      "epoch": 0.9157509157509157,
+      "grad_norm": 2.21875,
+      "learning_rate": 1.1851899215855653e-05,
+      "loss": 0.8985552215576171,
+      "mean_token_accuracy": 0.7717998087406158,
+      "num_tokens": 7700481.0,
+      "step": 1250
+    },
+    {
+      "entropy": 0.8970145070552826,
+      "epoch": 0.9340659340659341,
+      "grad_norm": 2.296875,
+      "learning_rate": 1.1559657292501044e-05,
+      "loss": 0.87716064453125,
+      "mean_token_accuracy": 0.7763631820678711,
+      "num_tokens": 7852495.0,
+      "step": 1275
+    },
+    {
+      "entropy": 0.9301983499526978,
+      "epoch": 0.9523809523809523,
+      "grad_norm": 2.09375,
+      "learning_rate": 1.1266043410114557e-05,
+      "loss": 0.907692642211914,
+      "mean_token_accuracy": 0.7715549063682556,
+      "num_tokens": 8001324.0,
+      "step": 1300
+    },
+    {
+      "entropy": 0.91446182847023,
+      "epoch": 0.9706959706959707,
+      "grad_norm": 2.359375,
+      "learning_rate": 1.0971315847370463e-05,
+      "loss": 0.8939350891113281,
+      "mean_token_accuracy": 0.7720365560054779,
+      "num_tokens": 8161385.0,
+      "step": 1325
+    },
+    {
+      "entropy": 0.9196098625659943,
+      "epoch": 0.989010989010989,
+      "grad_norm": 2.109375,
+      "learning_rate": 1.06757338625966e-05,
+      "loss": 0.89285888671875,
+      "mean_token_accuracy": 0.7721686267852783,
+      "num_tokens": 8315619.0,
+      "step": 1350
+    },
+    {
+      "entropy": 0.9019503581523896,
+      "epoch": 1.0073260073260073,
+      "grad_norm": 1.8828125,
+      "learning_rate": 1.0379557465716696e-05,
+      "loss": 0.853975830078125,
+      "mean_token_accuracy": 0.7786379647254944,
+      "num_tokens": 8474116.0,
+      "step": 1375
+    },
+    {
+      "entropy": 0.8720257365703583,
+      "epoch": 1.0256410256410255,
+      "grad_norm": 2.140625,
+      "learning_rate": 1.0083047189531566e-05,
+      "loss": 0.8370319366455078,
+      "mean_token_accuracy": 0.7838277363777161,
+      "num_tokens": 8633284.0,
+      "step": 1400
+    },
+    {
+      "epoch": 1.0256410256410255,
+      "eval_entropy": 0.8584034711950355,
+      "eval_loss": 0.9003849029541016,
+      "eval_mean_token_accuracy": 0.7730862986710336,
+      "eval_num_tokens": 8633284.0,
+      "eval_runtime": 8.0586,
+      "eval_samples_per_second": 285.285,
+      "eval_steps_per_second": 17.869,
+      "step": 1400
+    },
+    {
+      "entropy": 0.8384023177623748,
+      "epoch": 1.043956043956044,
+      "grad_norm": 2.15625,
+      "learning_rate": 9.786463860540352e-06,
+      "loss": 0.7985468292236328,
+      "mean_token_accuracy": 0.7922709083557129,
+      "num_tokens": 8785152.0,
+      "step": 1425
+    },
+    {
+      "entropy": 0.848993946313858,
+      "epoch": 1.0622710622710623,
+      "grad_norm": 2.234375,
+      "learning_rate": 9.49006836950339e-06,
+      "loss": 0.8184638977050781,
+      "mean_token_accuracy": 0.7883424413204193,
+      "num_tokens": 8936429.0,
+      "step": 1450
+    },
+    {
+      "entropy": 0.8467873334884644,
+      "epoch": 1.0805860805860805,
+      "grad_norm": 2.140625,
+      "learning_rate": 9.194121441948596e-06,
+      "loss": 0.8056396484375,
+      "mean_token_accuracy": 0.7903980255126953,
+      "num_tokens": 9094633.0,
+      "step": 1475
+    },
+    {
+      "entropy": 0.8402684187889099,
+      "epoch": 1.098901098901099,
+      "grad_norm": 2.109375,
+      "learning_rate": 8.898883408823124e-06,
+      "loss": 0.8085630798339843,
+      "mean_token_accuracy": 0.7898635935783386,
+      "num_tokens": 9246323.0,
+      "step": 1500
+    },
+    {
+      "entropy": 0.8303942859172821,
+      "epoch": 1.1172161172161172,
+      "grad_norm": 2.0625,
+      "learning_rate": 8.604613977492212e-06,
+      "loss": 0.7932289123535157,
+      "mean_token_accuracy": 0.7939286470413208,
+      "num_tokens": 9395722.0,
+      "step": 1525
+    },
+    {
+      "entropy": 0.8481007432937622,
+      "epoch": 1.1355311355311355,
+      "grad_norm": 2.109375,
+      "learning_rate": 8.3115720032865e-06,
+      "loss": 0.8194800567626953,
+      "mean_token_accuracy": 0.7893430662155151,
+      "num_tokens": 9547484.0,
+      "step": 1550
+    },
+    {
+      "entropy": 0.8354680657386779,
+      "epoch": 1.1538461538461537,
+      "grad_norm": 2.265625,
+      "learning_rate": 8.02001526179883e-06,
+      "loss": 0.8005447387695312,
+      "mean_token_accuracy": 0.7913288366794586,
+      "num_tokens": 9699959.0,
+      "step": 1575
+    },
+    {
+      "entropy": 0.8411949121952057,
+      "epoch": 1.1721611721611722,
+      "grad_norm": 2.125,
+      "learning_rate": 7.730200222130932e-06,
+      "loss": 0.810956039428711,
+      "mean_token_accuracy": 0.7901583468914032,
+      "num_tokens": 9851149.0,
+      "step": 1600
+    },
+    {
+      "epoch": 1.1721611721611722,
+      "eval_entropy": 0.8544707364506192,
+      "eval_loss": 0.8998146057128906,
+      "eval_mean_token_accuracy": 0.7731748426126109,
+      "eval_num_tokens": 9851149.0,
+      "eval_runtime": 8.0715,
+      "eval_samples_per_second": 284.828,
+      "eval_steps_per_second": 17.84,
+      "step": 1600
+    },
+    {
+      "entropy": 0.8444626975059509,
+      "epoch": 1.1904761904761905,
+      "grad_norm": 2.328125,
+      "learning_rate": 7.442381821289248e-06,
+      "loss": 0.8108631896972657,
+      "mean_token_accuracy": 0.7912653696537018,
+      "num_tokens": 9998793.0,
+      "step": 1625
+    },
+    {
+      "entropy": 0.8223755669593811,
+      "epoch": 1.2087912087912087,
+      "grad_norm": 2.09375,
+      "learning_rate": 7.1568132399285794e-06,
+      "loss": 0.7869499969482422,
+      "mean_token_accuracy": 0.7936726534366607,
+      "num_tokens": 10152752.0,
+      "step": 1650
+    },
+    {
+      "entropy": 0.8130457592010498,
+      "epoch": 1.2271062271062272,
+      "grad_norm": 1.875,
+      "learning_rate": 6.8737456796406594e-06,
+      "loss": 0.78310546875,
+      "mean_token_accuracy": 0.7966435635089875,
+      "num_tokens": 10308001.0,
+      "step": 1675
+    },
+    {
+      "entropy": 0.8471330416202545,
+      "epoch": 1.2454212454212454,
+      "grad_norm": 2.046875,
+      "learning_rate": 6.593428141983595e-06,
+      "loss": 0.8101668548583985,
+      "mean_token_accuracy": 0.7890221488475799,
+      "num_tokens": 10459655.0,
+      "step": 1700
+    },
+    {
+      "entropy": 0.8247973656654358,
+      "epoch": 1.2637362637362637,
+      "grad_norm": 2.1875,
+      "learning_rate": 6.316107209446662e-06,
+      "loss": 0.796153564453125,
+      "mean_token_accuracy": 0.7926836657524109,
+      "num_tokens": 10615183.0,
+      "step": 1725
+    },
+    {
+      "entropy": 0.8476881444454193,
+      "epoch": 1.282051282051282,
+      "grad_norm": 2.1875,
+      "learning_rate": 6.042026828542942e-06,
+      "loss": 0.8173642730712891,
+      "mean_token_accuracy": 0.7872111082077027,
+      "num_tokens": 10774019.0,
+      "step": 1750
+    },
+    {
+      "entropy": 0.8313502633571624,
+      "epoch": 1.3003663003663004,
+      "grad_norm": 2.0,
+      "learning_rate": 5.7714280952207955e-06,
+      "loss": 0.8023670959472656,
+      "mean_token_accuracy": 0.7926879894733428,
+      "num_tokens": 10926453.0,
+      "step": 1775
+    },
+    {
+      "entropy": 0.8281714594364167,
+      "epoch": 1.3186813186813187,
+      "grad_norm": 2.296875,
+      "learning_rate": 5.504549042782779e-06,
+      "loss": 0.7919468688964844,
+      "mean_token_accuracy": 0.7943351435661316,
+      "num_tokens": 11076817.0,
+      "step": 1800
+    },
+    {
+      "epoch": 1.3186813186813187,
+      "eval_entropy": 0.8518242807024055,
+      "eval_loss": 0.8994762897491455,
+      "eval_mean_token_accuracy": 0.7733377065095637,
+      "eval_num_tokens": 11076817.0,
+      "eval_runtime": 8.0691,
+      "eval_samples_per_second": 284.912,
+      "eval_steps_per_second": 17.846,
+      "step": 1800
+    },
+    {
+      "entropy": 0.8253132402896881,
+      "epoch": 1.3369963369963371,
+      "grad_norm": 2.09375,
+      "learning_rate": 5.241624432498673e-06,
+      "loss": 0.7934477996826171,
+      "mean_token_accuracy": 0.7905769181251526,
+      "num_tokens": 11235359.0,
+      "step": 1825
+    },
+    {
+      "entropy": 0.8527077162265777,
+      "epoch": 1.3553113553113554,
+      "grad_norm": 2.25,
+      "learning_rate": 4.982885547096754e-06,
+      "loss": 0.8256228637695312,
+      "mean_token_accuracy": 0.7879673659801483,
+      "num_tokens": 11385249.0,
+      "step": 1850
+    },
+    {
+      "entropy": 0.8239582288265228,
+      "epoch": 1.3736263736263736,
+      "grad_norm": 2.09375,
+      "learning_rate": 4.728559987314975e-06,
+      "loss": 0.7923411560058594,
+      "mean_token_accuracy": 0.7940970385074615,
+      "num_tokens": 11539970.0,
+      "step": 1875
+    },
+    {
+      "entropy": 0.8423762857913971,
+      "epoch": 1.3919413919413919,
+      "grad_norm": 2.1875,
+      "learning_rate": 4.478871471691031e-06,
+      "loss": 0.8075601196289063,
+      "mean_token_accuracy": 0.7895544397830964,
+      "num_tokens": 11694684.0,
+      "step": 1900
+    },
+    {
+      "entropy": 0.8148546552658081,
+      "epoch": 1.4102564102564101,
+      "grad_norm": 2.125,
+      "learning_rate": 4.234039639767406e-06,
+      "loss": 0.7813822937011718,
+      "mean_token_accuracy": 0.7967073798179627,
+      "num_tokens": 11850074.0,
+      "step": 1925
+    },
+    {
+      "entropy": 0.840382571220398,
+      "epoch": 1.4285714285714286,
+      "grad_norm": 2.09375,
+      "learning_rate": 3.994279858884551e-06,
+      "loss": 0.8069898986816406,
+      "mean_token_accuracy": 0.7921997106075287,
+      "num_tokens": 12004729.0,
+      "step": 1950
+    },
+    {
+      "entropy": 0.8522861528396607,
+      "epoch": 1.4468864468864469,
+      "grad_norm": 2.03125,
+      "learning_rate": 3.7598030347321e-06,
+      "loss": 0.8208354187011718,
+      "mean_token_accuracy": 0.789222640991211,
+      "num_tokens": 12157974.0,
+      "step": 1975
+    },
+    {
+      "entropy": 0.8414350926876069,
+      "epoch": 1.4652014652014653,
+      "grad_norm": 2.078125,
+      "learning_rate": 3.530815425824797e-06,
+      "loss": 0.8093458557128906,
+      "mean_token_accuracy": 0.7892864966392517,
+      "num_tokens": 12306150.0,
+      "step": 2000
+    },
+    {
+      "epoch": 1.4652014652014653,
+      "eval_entropy": 0.8502832547658019,
+      "eval_loss": 0.8990073204040527,
+      "eval_mean_token_accuracy": 0.7732960196832815,
+      "eval_num_tokens": 12306150.0,
+      "eval_runtime": 8.068,
+      "eval_samples_per_second": 284.952,
+      "eval_steps_per_second": 17.848,
+      "step": 2000
+    },
+    {
+      "entropy": 0.8521664845943451,
+      "epoch": 1.4835164835164836,
+      "grad_norm": 2.0625,
+      "learning_rate": 3.307518462066344e-06,
+      "loss": 0.8166194915771484,
+      "mean_token_accuracy": 0.785648273229599,
+      "num_tokens": 12456962.0,
+      "step": 2025
+    },
+    {
+      "entropy": 0.8251003730297088,
+      "epoch": 1.5018315018315018,
+      "grad_norm": 2.203125,
+      "learning_rate": 3.0901085675607266e-06,
+      "loss": 0.7916029357910156,
+      "mean_token_accuracy": 0.7955028152465821,
+      "num_tokens": 12606544.0,
+      "step": 2050
+    },
+    {
+      "entropy": 0.8439912271499633,
+      "epoch": 1.52014652014652,
+      "grad_norm": 2.203125,
+      "learning_rate": 2.8787769878269667e-06,
+      "loss": 0.811083755493164,
+      "mean_token_accuracy": 0.7906605911254883,
+      "num_tokens": 12763473.0,
+      "step": 2075
+    },
+    {
+      "entropy": 0.8612873029708862,
+      "epoch": 1.5384615384615383,
+      "grad_norm": 2.21875,
+      "learning_rate": 2.673709621569167e-06,
+      "loss": 0.834039077758789,
+      "mean_token_accuracy": 0.786132972240448,
+      "num_tokens": 12917918.0,
+      "step": 2100
+    },
+    {
+      "entropy": 0.832386200428009,
+      "epoch": 1.5567765567765568,
+      "grad_norm": 2.15625,
+      "learning_rate": 2.475086857149982e-06,
+      "loss": 0.7950384521484375,
+      "mean_token_accuracy": 0.7901384067535401,
+      "num_tokens": 13072296.0,
+      "step": 2125
+    },
+    {
+      "entropy": 0.8401387500762939,
+      "epoch": 1.575091575091575,
+      "grad_norm": 2.140625,
+      "learning_rate": 2.2830834139112136e-06,
+      "loss": 0.8060906219482422,
+      "mean_token_accuracy": 0.7896384346485138,
+      "num_tokens": 13226077.0,
+      "step": 2150
+    },
+    {
+      "entropy": 0.8430456602573395,
+      "epoch": 1.5934065934065935,
+      "grad_norm": 2.28125,
+      "learning_rate": 2.097868188481217e-06,
+      "loss": 0.8120394134521485,
+      "mean_token_accuracy": 0.7886982035636901,
+      "num_tokens": 13379631.0,
+      "step": 2175
+    },
+    {
+      "entropy": 0.8341074979305267,
+      "epoch": 1.6117216117216118,
+      "grad_norm": 2.28125,
+      "learning_rate": 1.919604106204298e-06,
+      "loss": 0.7923731231689453,
+      "mean_token_accuracy": 0.7932259738445282,
+      "num_tokens": 13537431.0,
+      "step": 2200
+    },
+    {
+      "epoch": 1.6117216117216118,
+      "eval_entropy": 0.8466265425086021,
+      "eval_loss": 0.8991615176200867,
+      "eval_mean_token_accuracy": 0.7732404433190823,
+      "eval_num_tokens": 13537431.0,
+      "eval_runtime": 8.0784,
+      "eval_samples_per_second": 284.585,
+      "eval_steps_per_second": 17.825,
+      "step": 2200
+    },
+    {
+      "entropy": 0.829889565706253,
+      "epoch": 1.63003663003663,
+      "grad_norm": 2.09375,
+      "learning_rate": 1.7484479778227281e-06,
+      "loss": 0.7963677215576171,
+      "mean_token_accuracy": 0.7930803084373474,
+      "num_tokens": 13696124.0,
+      "step": 2225
+    },
+    {
+      "entropy": 0.8456670987606049,
+      "epoch": 1.6483516483516483,
+      "grad_norm": 2.125,
+      "learning_rate": 1.5845503615375534e-06,
+      "loss": 0.8171965789794922,
+      "mean_token_accuracy": 0.78998281955719,
+      "num_tokens": 13850353.0,
+      "step": 2250
+    },
+    {
+      "entropy": 0.834287303686142,
+      "epoch": 1.6666666666666665,
+      "grad_norm": 2.203125,
+      "learning_rate": 1.4280554305694205e-06,
+      "loss": 0.8009786987304688,
+      "mean_token_accuracy": 0.7929751622676849,
+      "num_tokens": 14002830.0,
+      "step": 2275
+    },
+    {
+      "entropy": 0.8210822236537934,
+      "epoch": 1.684981684981685,
+      "grad_norm": 2.171875,
+      "learning_rate": 1.2791008463360077e-06,
+      "loss": 0.7856560516357421,
+      "mean_token_accuracy": 0.7937294292449951,
+      "num_tokens": 14159802.0,
+      "step": 2300
+    },
+    {
+      "entropy": 0.8226444447040557,
+      "epoch": 1.7032967032967035,
+      "grad_norm": 2.09375,
+      "learning_rate": 1.1378176373575977e-06,
+      "loss": 0.7973332977294922,
+      "mean_token_accuracy": 0.7935956382751465,
+      "num_tokens": 14315490.0,
+      "step": 2325
+    },
+    {
+      "entropy": 0.8418577921390533,
+      "epoch": 1.7216117216117217,
+      "grad_norm": 2.203125,
+      "learning_rate": 1.004330083997278e-06,
+      "loss": 0.8100568389892578,
+      "mean_token_accuracy": 0.79057302236557,
+      "num_tokens": 14471457.0,
+      "step": 2350
+    },
+    {
+      "entropy": 0.8569370365142822,
+      "epoch": 1.73992673992674,
+      "grad_norm": 2.296875,
+      "learning_rate": 8.787556091372207e-07,
+      "loss": 0.8271647644042969,
+      "mean_token_accuracy": 0.783940087556839,
+      "num_tokens": 14627084.0,
+      "step": 2375
+    },
+    {
+      "entropy": 0.8276915633678437,
+      "epoch": 1.7582417582417582,
+      "grad_norm": 2.171875,
+      "learning_rate": 7.612046748871327e-07,
+      "loss": 0.8014369201660156,
+      "mean_token_accuracy": 0.790752956867218,
+      "num_tokens": 14772996.0,
+      "step": 2400
+    },
+    {
+      "epoch": 1.7582417582417582,
+      "eval_entropy": 0.8465850080053011,
+      "eval_loss": 0.899135172367096,
+      "eval_mean_token_accuracy": 0.7732518298758401,
+      "eval_num_tokens": 14772996.0,
+      "eval_runtime": 8.0715,
+      "eval_samples_per_second": 284.83,
+      "eval_steps_per_second": 17.841,
+      "step": 2400
+    },
+    {
+      "entropy": 0.8189385521411896,
+      "epoch": 1.7765567765567765,
+      "grad_norm": 2.09375,
+      "learning_rate": 6.517806854158204e-07,
+      "loss": 0.7890790557861328,
+      "mean_token_accuracy": 0.7950505125522613,
+      "num_tokens": 14926511.0,
+      "step": 2425
+    },
+    {
+      "entropy": 0.8378893113136292,
+      "epoch": 1.7948717948717947,
+      "grad_norm": 2.359375,
+      "learning_rate": 5.505798959912723e-07,
+      "loss": 0.8117930603027343,
+      "mean_token_accuracy": 0.7887657046318054,
+      "num_tokens": 15081687.0,
+      "step": 2450
+    },
+    {
+      "entropy": 0.8320703601837158,
+      "epoch": 1.8131868131868132,
+      "grad_norm": 2.140625,
+      "learning_rate": 4.576913283093098e-07,
+      "loss": 0.8038975524902344,
+      "mean_token_accuracy": 0.7904533529281617,
+      "num_tokens": 15236278.0,
+      "step": 2475
+    },
+    {
+      "entropy": 0.8401167035102844,
+      "epoch": 1.8315018315018317,
+      "grad_norm": 2.234375,
+      "learning_rate": 3.7319669218529475e-07,
+      "loss": 0.8033819580078125,
+      "mean_token_accuracy": 0.7910889124870301,
+      "num_tokens": 15391169.0,
+      "step": 2500
+    },
+    {
+      "entropy": 0.858134560585022,
+      "epoch": 1.84981684981685,
+      "grad_norm": 2.359375,
+      "learning_rate": 2.971703136777315e-07,
+      "loss": 0.8318187713623046,
+      "mean_token_accuracy": 0.784791008234024,
+      "num_tokens": 15548066.0,
+      "step": 2525
+    },
+    {
+      "entropy": 0.8192259454727173,
+      "epoch": 1.8681318681318682,
+      "grad_norm": 1.9453125,
+      "learning_rate": 2.2967906970705722e-07,
+      "loss": 0.7859656524658203,
+      "mean_token_accuracy": 0.7967568933963776,
+      "num_tokens": 15699746.0,
+      "step": 2550
+    },
+    {
+      "entropy": 0.8540216779708862,
+      "epoch": 1.8864468864468864,
+      "grad_norm": 2.171875,
+      "learning_rate": 1.7078232922706495e-07,
+      "loss": 0.8298044586181641,
+      "mean_token_accuracy": 0.7818279254436493,
+      "num_tokens": 15856690.0,
+      "step": 2575
+    },
+    {
+      "entropy": 0.8150307929515839,
+      "epoch": 1.9047619047619047,
+      "grad_norm": 2.109375,
+      "learning_rate": 1.2053190100077106e-07,
+      "loss": 0.7887921905517579,
+      "mean_token_accuracy": 0.7932768344879151,
+      "num_tokens": 16013738.0,
+      "step": 2600
+    },
+    {
+      "epoch": 1.9047619047619047,
+      "eval_entropy": 0.8467293654878935,
+      "eval_loss": 0.8991142511367798,
+      "eval_mean_token_accuracy": 0.7734044496383932,
+      "eval_num_tokens": 16013738.0,
+      "eval_runtime": 8.0684,
+      "eval_samples_per_second": 284.939,
+      "eval_steps_per_second": 17.847,
+      "step": 2600
+    },
+    {
+      "entropy": 0.8457963478565216,
+      "epoch": 1.9230769230769231,
+      "grad_norm": 2.1875,
+      "learning_rate": 7.897198802661266e-08,
+      "loss": 0.8114620971679688,
+      "mean_token_accuracy": 0.7876574242115021,
+      "num_tokens": 16168732.0,
+      "step": 2625
+    },
+    {
+      "entropy": 0.8315440094470978,
+      "epoch": 1.9413919413919414,
+      "grad_norm": 2.09375,
+      "learning_rate": 4.613914865509439e-08,
+      "loss": 0.8015847778320313,
+      "mean_token_accuracy": 0.7910509896278382,
+      "num_tokens": 16319610.0,
+      "step": 2650
+    },
+    {
+      "entropy": 0.8333329391479493,
+      "epoch": 1.9597069597069599,
+      "grad_norm": 2.1875,
+      "learning_rate": 2.2062264430087187e-08,
+      "loss": 0.8026151275634765,
+      "mean_token_accuracy": 0.7906608557701111,
+      "num_tokens": 16484108.0,
+      "step": 2675
+    },
+    {
+      "entropy": 0.8297130846977234,
+      "epoch": 1.978021978021978,
+      "grad_norm": 2.0625,
+      "learning_rate": 6.762514683058552e-09,
+      "loss": 0.7984342193603515,
+      "mean_token_accuracy": 0.7927398717403412,
+      "num_tokens": 16637988.0,
+      "step": 2700
+    },
+    {
+      "entropy": 0.8448534452915192,
+      "epoch": 1.9963369963369964,
+      "grad_norm": 2.1875,
+      "learning_rate": 2.5335790258207426e-10,
+      "loss": 0.8189653015136719,
+      "mean_token_accuracy": 0.7890227103233337,
+      "num_tokens": 16789803.0,
+      "step": 2725
+    },
+    {
+      "epoch": 2.0,
+      "eval_entropy": 0.8461464825603697,
+      "eval_loss": 0.8991644978523254,
+      "eval_mean_token_accuracy": 0.7732141961654028,
+      "eval_num_tokens": 16819082.0,
+      "eval_runtime": 8.0725,
+      "eval_samples_per_second": 284.796,
+      "eval_steps_per_second": 17.838,
+      "step": 2730
+    }
+  ],
+  "logging_steps": 25,
+  "max_steps": 2730,
+  "num_input_tokens_seen": 0,
+  "num_train_epochs": 2,
+  "save_steps": 200,
+  "stateful_callbacks": {
+    "TrainerControl": {
+      "args": {
+        "should_epoch_stop": false,
+        "should_evaluate": false,
+        "should_log": false,
+        "should_save": true,
+        "should_training_stop": true
+      },
+      "attributes": {}
+    }
+  },
+  "total_flos": 6.805484445629952e+16,
+  "train_batch_size": 16,
+  "trial_name": null,
+  "trial_params": null
+}

checkpoint-2730/training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1f3eec82a82cc2cb4f2c124367cd9fb480a301c5cdfda379c7339e6e590e5bd3
+size 5585

config.json

@@ -0,0 +1,60 @@
+{
+  "architectures": [
+    "Lfm2ForCausalLM"
+  ],
+  "block_auto_adjust_ff_dim": true,
+  "block_dim": 1024,
+  "block_ffn_dim_multiplier": 1.0,
+  "block_mlp_init_scale": 1.0,
+  "block_multiple_of": 256,
+  "block_norm_eps": 1e-05,
+  "block_out_init_scale": 1.0,
+  "block_use_swiglu": true,
+  "block_use_xavier_init": true,
+  "bos_token_id": 1,
+  "conv_L_cache": 3,
+  "conv_bias": false,
+  "conv_dim": 1024,
+  "conv_use_xavier_init": true,
+  "dtype": "bfloat16",
+  "eos_token_id": 7,
+  "full_attn_idxs": null,
+  "hidden_size": 1024,
+  "initializer_range": 0.02,
+  "intermediate_size": 6656,
+  "layer_types": [
+    "conv",
+    "conv",
+    "full_attention",
+    "conv",
+    "conv",
+    "full_attention",
+    "conv",
+    "conv",
+    "full_attention",
+    "conv",
+    "full_attention",
+    "conv",
+    "full_attention",
+    "conv",
+    "full_attention",
+    "conv"
+  ],
+  "max_position_embeddings": 128000,
+  "model_type": "lfm2",
+  "norm_eps": 1e-05,
+  "num_attention_heads": 16,
+  "num_heads": 16,
+  "num_hidden_layers": 16,
+  "num_key_value_heads": 8,
+  "pad_token_id": 0,
+  "rope_parameters": {
+    "rope_theta": 1000000.0,
+    "rope_type": "default"
+  },
+  "tie_word_embeddings": true,
+  "transformers_version": "5.5.0",
+  "use_cache": false,
+  "use_pos_enc": true,
+  "vocab_size": 65536
+}

generation_config.json

@@ -0,0 +1,9 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 1,
+  "eos_token_id": [
+    7
+  ],
+  "pad_token_id": 0,
+  "transformers_version": "5.5.0"
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9393f975e13b7b4f0ddbb9081a5ab9986b609333524c504e9dcd68109beeaa02
+size 708984464

tokenizer_config.json

@@ -0,0 +1,21 @@
+{
+  "backend": "tokenizers",
+  "bos_token": "<|startoftext|>",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|im_end|>",
+  "extra_special_tokens": [],
+  "is_local": false,
+  "legacy": false,
+  "model_input_names": [
+    "input_ids",
+    "attention_mask"
+  ],
+  "model_max_length": 1000000000000000019884624838656,
+  "model_specific_special_tokens": {},
+  "pad_token": "<|pad|>",
+  "sp_model_kwargs": {},
+  "spaces_between_special_tokens": false,
+  "tokenizer_class": "TokenizersBackend",
+  "use_default_system_prompt": false,
+  "use_fast": true
+}

training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1f3eec82a82cc2cb4f2c124367cd9fb480a301c5cdfda379c7339e6e590e5bd3
+size 5585