Instructions to use modrill/qwen3_4b_probe_lingcoder_lora_1g with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use modrill/qwen3_4b_probe_lingcoder_lora_1g with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="modrill/qwen3_4b_probe_lingcoder_lora_1g")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoModel
model = AutoModel.from_pretrained("modrill/qwen3_4b_probe_lingcoder_lora_1g", dtype="auto")

Notebooks
Google Colab
Kaggle
Local Apps

vLLM

How to use modrill/qwen3_4b_probe_lingcoder_lora_1g with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "modrill/qwen3_4b_probe_lingcoder_lora_1g"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "modrill/qwen3_4b_probe_lingcoder_lora_1g",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/modrill/qwen3_4b_probe_lingcoder_lora_1g

SGLang

How to use modrill/qwen3_4b_probe_lingcoder_lora_1g with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "modrill/qwen3_4b_probe_lingcoder_lora_1g" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "modrill/qwen3_4b_probe_lingcoder_lora_1g",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "modrill/qwen3_4b_probe_lingcoder_lora_1g" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "modrill/qwen3_4b_probe_lingcoder_lora_1g",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use modrill/qwen3_4b_probe_lingcoder_lora_1g with Docker Model Runner:
```
docker model run hf.co/modrill/qwen3_4b_probe_lingcoder_lora_1g
```

modrill commited on 10 days ago

Commit

a72802b

verified ·

1 Parent(s): 2889a4c

Add files using upload-large-folder tool

Browse files

Files changed (19) hide show

.gitattributes +2 -0
README.md +56 -10
adapter_config.json +48 -0
adapter_model.safetensors +3 -0
chat_template.jinja +12 -0
checkpoint-256/README.md +209 -0
checkpoint-256/adapter_config.json +48 -0
checkpoint-256/adapter_model.safetensors +3 -0
checkpoint-256/chat_template.jinja +12 -0
checkpoint-256/optimizer.pt +3 -0
checkpoint-256/rng_state.pth +3 -0
checkpoint-256/scheduler.pt +3 -0
checkpoint-256/tokenizer.json +3 -0
checkpoint-256/tokenizer_config.json +30 -0
checkpoint-256/trainer_state.json +2594 -0
checkpoint-256/training_args.bin +3 -0
tokenizer.json +3 -0
tokenizer_config.json +30 -0
training_args.bin +3 -0

.gitattributes CHANGED Viewed

@@ -33,3 +33,5 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text

 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+checkpoint-256/tokenizer.json filter=lfs diff=lfs merge=lfs -text
+tokenizer.json filter=lfs diff=lfs merge=lfs -text

README.md CHANGED Viewed

@@ -1,16 +1,62 @@
 ---
-license: cc-by-nc-4.0
 tags:
-- qwen3
-- mhm
-- text-generation
-library_name: transformers
 ---
-# qwen3_4b_probe_lingcoder_lora_1g
-Auto-uploaded by watcher.
-- Source path: `trl/models/qwen3-4b-probe-lingcoder-lora-1g`
-- Uploaded at: `2026-05-20T05:45:25.039370`
-- Visibility: `public`

 ---
+base_model: Qwen/Qwen3-4B-Base
+library_name: peft
+model_name: qwen3-4b-probe-lingcoder-lora-1g
 tags:
+- base_model:adapter:Qwen/Qwen3-4B-Base
+- lora
+- sft
+- transformers
+- trl
+licence: license
+pipeline_tag: text-generation
 ---
+# Model Card for qwen3-4b-probe-lingcoder-lora-1g
+This model is a fine-tuned version of [Qwen/Qwen3-4B-Base](https://huggingface.co/Qwen/Qwen3-4B-Base).
+It has been trained using [TRL](https://github.com/huggingface/trl).
+## Quick start
+```python
+from transformers import pipeline
+question = "If you had a time machine, but could only go to the past or the future once and never return, which would you choose and why?"
+generator = pipeline("text-generation", model="None", device="cuda")
+output = generator([{"role": "user", "content": question}], max_new_tokens=128, return_full_text=False)[0]
+print(output["generated_text"])
+```
+## Training procedure
+This model was trained with SFT.
+### Framework versions
+- PEFT 0.19.1
+- TRL: 1.5.0.dev0
+- Transformers: 5.8.0
+- Pytorch: 2.11.0
+- Datasets: 4.8.5
+- Tokenizers: 0.22.2
+## Citations
+Cite TRL as:
+```bibtex
+@software{vonwerra2020trl,
+  title   = {{TRL: Transformers Reinforcement Learning}},
+  author  = {von Werra, Leandro and Belkada, Younes and Tunstall, Lewis and Beeching, Edward and Thrush, Tristan and Lambert, Nathan and Huang, Shengyi and Rasul, Kashif and Gallouédec, Quentin},
+  license = {Apache-2.0},
+  url     = {https://github.com/huggingface/trl},
+  year    = {2020}
+}
+```

adapter_config.json ADDED Viewed

	@@ -0,0 +1,48 @@

+{
+  "alora_invocation_tokens": null,
+  "alpha_pattern": {},
+  "arrow_config": null,
+  "auto_mapping": null,
+  "base_model_name_or_path": "Qwen/Qwen3-4B-Base",
+  "bias": "none",
+  "corda_config": null,
+  "ensure_weight_tying": false,
+  "eva_config": null,
+  "exclude_modules": null,
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 64,
+  "lora_bias": false,
+  "lora_dropout": 0.05,
+  "lora_ga_config": null,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "peft_version": "0.19.1",
+  "qalora_group_size": 16,
+  "r": 32,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "k_proj",
+    "o_proj",
+    "up_proj",
+    "gate_proj",
+    "q_proj",
+    "down_proj",
+    "v_proj"
+  ],
+  "target_parameters": null,
+  "task_type": "CAUSAL_LM",
+  "trainable_token_indices": null,
+  "use_bdlora": null,
+  "use_dora": false,
+  "use_qalora": false,
+  "use_rslora": false
+}

adapter_model.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:9e00bad0b82f58ffd38ca561f7b9cf14f419427c4322937e2a3f88df288688b2
+size 264308896

chat_template.jinja ADDED Viewed

	@@ -0,0 +1,12 @@

+{%- for message in messages %}
+{%- if message.role == "user" %}
+{{- '<|im_start|>user\n' + message.content + '<|im_end|>\n' }}
+{%- elif message.role == "system" %}
+{{- '<|im_start|>system\n' + message.content + '<|im_end|>\n' }}
+{%- elif message.role == "assistant" %}
+{{- '<|im_start|>assistant\n' }}{% generation %}{{ message.content }}{% endgeneration %}{{ '<|im_end|>\n' }}
+{%- endif %}
+{%- endfor %}
+{%- if add_generation_prompt %}
+{{- '<|im_start|>assistant\n' }}
+{%- endif %}

checkpoint-256/README.md ADDED Viewed

	@@ -0,0 +1,209 @@

+---
+base_model: Qwen/Qwen3-4B-Base
+library_name: peft
+pipeline_tag: text-generation
+tags:
+- base_model:adapter:Qwen/Qwen3-4B-Base
+- lora
+- sft
+- transformers
+- trl
+---
+# Model Card for Model ID
+<!-- Provide a quick summary of what the model is/does. -->
+## Model Details
+### Model Description
+<!-- Provide a longer summary of what this model is. -->
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+### Model Sources [optional]
+<!-- Provide the basic links for the model. -->
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+## Uses
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+### Direct Use
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+[More Information Needed]
+### Downstream Use [optional]
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+[More Information Needed]
+### Out-of-Scope Use
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+[More Information Needed]
+## Bias, Risks, and Limitations
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+[More Information Needed]
+### Recommendations
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+## How to Get Started with the Model
+Use the code below to get started with the model.
+[More Information Needed]
+## Training Details
+### Training Data
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+[More Information Needed]
+### Training Procedure
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+#### Preprocessing [optional]
+[More Information Needed]
+#### Training Hyperparameters
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+#### Speeds, Sizes, Times [optional]
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+[More Information Needed]
+## Evaluation
+<!-- This section describes the evaluation protocols and provides the results. -->
+### Testing Data, Factors & Metrics
+#### Testing Data
+<!-- This should link to a Dataset Card if possible. -->
+[More Information Needed]
+#### Factors
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+[More Information Needed]
+#### Metrics
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+[More Information Needed]
+### Results
+[More Information Needed]
+#### Summary
+## Model Examination [optional]
+<!-- Relevant interpretability work for the model goes here -->
+[More Information Needed]
+## Environmental Impact
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+## Technical Specifications [optional]
+### Model Architecture and Objective
+[More Information Needed]
+### Compute Infrastructure
+[More Information Needed]
+#### Hardware
+[More Information Needed]
+#### Software
+[More Information Needed]
+## Citation [optional]
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+**BibTeX:**
+[More Information Needed]
+**APA:**
+[More Information Needed]
+## Glossary [optional]
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+[More Information Needed]
+## More Information [optional]
+[More Information Needed]
+## Model Card Authors [optional]
+[More Information Needed]
+## Model Card Contact
+[More Information Needed]
+### Framework versions
+- PEFT 0.19.1

checkpoint-256/adapter_config.json ADDED Viewed

	@@ -0,0 +1,48 @@

+{
+  "alora_invocation_tokens": null,
+  "alpha_pattern": {},
+  "arrow_config": null,
+  "auto_mapping": null,
+  "base_model_name_or_path": "Qwen/Qwen3-4B-Base",
+  "bias": "none",
+  "corda_config": null,
+  "ensure_weight_tying": false,
+  "eva_config": null,
+  "exclude_modules": null,
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 64,
+  "lora_bias": false,
+  "lora_dropout": 0.05,
+  "lora_ga_config": null,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "peft_version": "0.19.1",
+  "qalora_group_size": 16,
+  "r": 32,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "k_proj",
+    "o_proj",
+    "up_proj",
+    "gate_proj",
+    "q_proj",
+    "down_proj",
+    "v_proj"
+  ],
+  "target_parameters": null,
+  "task_type": "CAUSAL_LM",
+  "trainable_token_indices": null,
+  "use_bdlora": null,
+  "use_dora": false,
+  "use_qalora": false,
+  "use_rslora": false
+}

checkpoint-256/adapter_model.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:9e00bad0b82f58ffd38ca561f7b9cf14f419427c4322937e2a3f88df288688b2
+size 264308896

checkpoint-256/chat_template.jinja ADDED Viewed

	@@ -0,0 +1,12 @@

+{%- for message in messages %}
+{%- if message.role == "user" %}
+{{- '<|im_start|>user\n' + message.content + '<|im_end|>\n' }}
+{%- elif message.role == "system" %}
+{{- '<|im_start|>system\n' + message.content + '<|im_end|>\n' }}
+{%- elif message.role == "assistant" %}
+{{- '<|im_start|>assistant\n' }}{% generation %}{{ message.content }}{% endgeneration %}{{ '<|im_end|>\n' }}
+{%- endif %}
+{%- endfor %}
+{%- if add_generation_prompt %}
+{{- '<|im_start|>assistant\n' }}
+{%- endif %}

checkpoint-256/optimizer.pt ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:c7693c8d3dd390c89aaf4c619635f1dbcd2162bb99e58ff6718ddb792cd4e641
+size 528915403

checkpoint-256/rng_state.pth ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:90a7e42456cdc1b276a17850dd27e2e254c15b3e9404b7f14df06aee71e9c332
+size 14709

checkpoint-256/scheduler.pt ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:b9796310e6c783b70334f842ffbcd2b6bad30a61ebe6dc7fd60aae80a8ddb82b
+size 1465

checkpoint-256/tokenizer.json ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:be75606093db2094d7cd20f3c2f385c212750648bd6ea4fb2bf507a6a4c55506
+size 11422650

checkpoint-256/tokenizer_config.json ADDED Viewed

	@@ -0,0 +1,30 @@

+{
+  "add_prefix_space": false,
+  "backend": "tokenizers",
+  "bos_token": null,
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|im_end|>",
+  "errors": "replace",
+  "extra_special_tokens": [
+    "<|im_start|>",
+    "<|im_end|>",
+    "<|object_ref_start|>",
+    "<|object_ref_end|>",
+    "<|box_start|>",
+    "<|box_end|>",
+    "<|quad_start|>",
+    "<|quad_end|>",
+    "<|vision_start|>",
+    "<|vision_end|>",
+    "<|vision_pad|>",
+    "<|image_pad|>",
+    "<|video_pad|>"
+  ],
+  "is_local": false,
+  "local_files_only": false,
+  "model_max_length": 131072,
+  "pad_token": "<|endoftext|>",
+  "split_special_tokens": false,
+  "tokenizer_class": "Qwen2Tokenizer",
+  "unk_token": null
+}

checkpoint-256/trainer_state.json ADDED Viewed

	@@ -0,0 +1,2594 @@

+{
+  "best_global_step": null,
+  "best_metric": null,
+  "best_model_checkpoint": null,
+  "epoch": 1.0,
+  "eval_steps": 500,
+  "global_step": 256,
+  "is_hyper_param_search": false,
+  "is_local_process_zero": true,
+  "is_world_process_zero": true,
+  "log_history": [
+    {
+      "entropy": 0.2590447962284088,
+      "epoch": 0.00390625,
+      "grad_norm": 0.7801761627197266,
+      "learning_rate": 0.0,
+      "loss": 0.2889120280742645,
+      "mean_token_accuracy": 0.9251968264579773,
+      "num_tokens": 614.0,
+      "step": 1
+    },
+    {
+      "entropy": 0.8471839427947998,
+      "epoch": 0.0078125,
+      "grad_norm": 1.0937414169311523,
+      "learning_rate": 7.692307692307694e-06,
+      "loss": 0.9662710428237915,
+      "mean_token_accuracy": 0.7279693484306335,
+      "num_tokens": 1246.0,
+      "step": 2
+    },
+    {
+      "entropy": 0.5400290489196777,
+      "epoch": 0.01171875,
+      "grad_norm": 0.5771386027336121,
+      "learning_rate": 1.5384615384615387e-05,
+      "loss": 0.6009750962257385,
+      "mean_token_accuracy": 0.8273224234580994,
+      "num_tokens": 2224.0,
+      "step": 3
+    },
+    {
+      "entropy": 0.5040320158004761,
+      "epoch": 0.015625,
+      "grad_norm": 0.5884291529655457,
+      "learning_rate": 2.307692307692308e-05,
+      "loss": 0.571037232875824,
+      "mean_token_accuracy": 0.8431817889213562,
+      "num_tokens": 3178.0,
+      "step": 4
+    },
+    {
+      "entropy": 0.7248499989509583,
+      "epoch": 0.01953125,
+      "grad_norm": 0.6174550652503967,
+      "learning_rate": 3.0769230769230774e-05,
+      "loss": 0.7390681505203247,
+      "mean_token_accuracy": 0.7871825695037842,
+      "num_tokens": 4063.0,
+      "step": 5
+    },
+    {
+      "entropy": 0.49000677466392517,
+      "epoch": 0.0234375,
+      "grad_norm": 0.6189038157463074,
+      "learning_rate": 3.846153846153846e-05,
+      "loss": 0.6010043621063232,
+      "mean_token_accuracy": 0.8243626356124878,
+      "num_tokens": 5183.0,
+      "step": 6
+    },
+    {
+      "entropy": 0.2592145800590515,
+      "epoch": 0.02734375,
+      "grad_norm": 0.4830591678619385,
+      "learning_rate": 4.615384615384616e-05,
+      "loss": 0.2511421740055084,
+      "mean_token_accuracy": 0.9138461351394653,
+      "num_tokens": 6572.0,
+      "step": 7
+    },
+    {
+      "entropy": 0.6752243638038635,
+      "epoch": 0.03125,
+      "grad_norm": 0.6484870314598083,
+      "learning_rate": 5.384615384615385e-05,
+      "loss": 0.7448326349258423,
+      "mean_token_accuracy": 0.7770800590515137,
+      "num_tokens": 7280.0,
+      "step": 8
+    },
+    {
+      "entropy": 0.44495999813079834,
+      "epoch": 0.03515625,
+      "grad_norm": 0.5341415405273438,
+      "learning_rate": 6.153846153846155e-05,
+      "loss": 0.517272412776947,
+      "mean_token_accuracy": 0.8297587037086487,
+      "num_tokens": 8114.0,
+      "step": 9
+    },
+    {
+      "entropy": 0.5511445999145508,
+      "epoch": 0.0390625,
+      "grad_norm": 0.5533967614173889,
+      "learning_rate": 6.923076923076924e-05,
+      "loss": 0.5668619871139526,
+      "mean_token_accuracy": 0.809568464756012,
+      "num_tokens": 9241.0,
+      "step": 10
+    },
+    {
+      "entropy": 0.5666453838348389,
+      "epoch": 0.04296875,
+      "grad_norm": 0.5970139503479004,
+      "learning_rate": 7.692307692307693e-05,
+      "loss": 0.5543555617332458,
+      "mean_token_accuracy": 0.8302631378173828,
+      "num_tokens": 10061.0,
+      "step": 11
+    },
+    {
+      "entropy": 0.6745113134384155,
+      "epoch": 0.046875,
+      "grad_norm": 0.7207919955253601,
+      "learning_rate": 8.461538461538461e-05,
+      "loss": 0.7640900611877441,
+      "mean_token_accuracy": 0.8009049892425537,
+      "num_tokens": 10774.0,
+      "step": 12
+    },
+    {
+      "entropy": 0.5185617804527283,
+      "epoch": 0.05078125,
+      "grad_norm": 0.5652928948402405,
+      "learning_rate": 9.230769230769232e-05,
+      "loss": 0.48071104288101196,
+      "mean_token_accuracy": 0.8530259132385254,
+      "num_tokens": 11515.0,
+      "step": 13
+    },
+    {
+      "entropy": 0.21858422458171844,
+      "epoch": 0.0546875,
+      "grad_norm": 0.5040489435195923,
+      "learning_rate": 0.0001,
+      "loss": 0.19600161910057068,
+      "mean_token_accuracy": 0.9303675293922424,
+      "num_tokens": 12415.0,
+      "step": 14
+    },
+    {
+      "entropy": 0.8073277473449707,
+      "epoch": 0.05859375,
+      "grad_norm": 0.7844629883766174,
+      "learning_rate": 9.999582149277187e-05,
+      "loss": 0.7193360328674316,
+      "mean_token_accuracy": 0.7882096171379089,
+      "num_tokens": 13173.0,
+      "step": 15
+    },
+    {
+      "entropy": 1.1454415321350098,
+      "epoch": 0.0625,
+      "grad_norm": 0.974055290222168,
+      "learning_rate": 9.998328666948438e-05,
+      "loss": 1.1323111057281494,
+      "mean_token_accuracy": 0.6666666865348816,
+      "num_tokens": 13920.0,
+      "step": 16
+    },
+    {
+      "entropy": 0.61671382188797,
+      "epoch": 0.06640625,
+      "grad_norm": 0.5854765772819519,
+      "learning_rate": 9.996239762521151e-05,
+      "loss": 0.5350141525268555,
+      "mean_token_accuracy": 0.8413524031639099,
+      "num_tokens": 14748.0,
+      "step": 17
+    },
+    {
+      "entropy": 0.7238249778747559,
+      "epoch": 0.0703125,
+      "grad_norm": 0.577200174331665,
+      "learning_rate": 9.993315785135416e-05,
+      "loss": 0.6484571695327759,
+      "mean_token_accuracy": 0.8147714138031006,
+      "num_tokens": 15659.0,
+      "step": 18
+    },
+    {
+      "entropy": 0.42435163259506226,
+      "epoch": 0.07421875,
+      "grad_norm": 0.7431618571281433,
+      "learning_rate": 9.989557223505661e-05,
+      "loss": 0.3732726573944092,
+      "mean_token_accuracy": 0.8780487775802612,
+      "num_tokens": 16469.0,
+      "step": 19
+    },
+    {
+      "entropy": 0.5280213356018066,
+      "epoch": 0.078125,
+      "grad_norm": 0.5703546404838562,
+      "learning_rate": 9.98496470583896e-05,
+      "loss": 0.47382473945617676,
+      "mean_token_accuracy": 0.8323615193367004,
+      "num_tokens": 17241.0,
+      "step": 20
+    },
+    {
+      "entropy": 0.2634253203868866,
+      "epoch": 0.08203125,
+      "grad_norm": 0.8636095523834229,
+      "learning_rate": 9.979538999730047e-05,
+      "loss": 0.2421087771654129,
+      "mean_token_accuracy": 0.9324324131011963,
+      "num_tokens": 17735.0,
+      "step": 21
+    },
+    {
+      "entropy": 0.7180085182189941,
+      "epoch": 0.0859375,
+      "grad_norm": 0.848982036113739,
+      "learning_rate": 9.973281012033007e-05,
+      "loss": 0.6361225247383118,
+      "mean_token_accuracy": 0.8064516186714172,
+      "num_tokens": 18390.0,
+      "step": 22
+    },
+    {
+      "entropy": 0.5791106224060059,
+      "epoch": 0.08984375,
+      "grad_norm": 0.5454062223434448,
+      "learning_rate": 9.966191788709716e-05,
+      "loss": 0.5845445394515991,
+      "mean_token_accuracy": 0.824435293674469,
+      "num_tokens": 19418.0,
+      "step": 23
+    },
+    {
+      "entropy": 0.4929169714450836,
+      "epoch": 0.09375,
+      "grad_norm": 0.5984290242195129,
+      "learning_rate": 9.958272514655006e-05,
+      "loss": 0.49152883887290955,
+      "mean_token_accuracy": 0.8545246124267578,
+      "num_tokens": 20349.0,
+      "step": 24
+    },
+    {
+      "entropy": 0.5289084315299988,
+      "epoch": 0.09765625,
+      "grad_norm": 0.597358763217926,
+      "learning_rate": 9.949524513498636e-05,
+      "loss": 0.5099250674247742,
+      "mean_token_accuracy": 0.8437103033065796,
+      "num_tokens": 21202.0,
+      "step": 25
+    },
+    {
+      "entropy": 0.43412598967552185,
+      "epoch": 0.1015625,
+      "grad_norm": 1.2732841968536377,
+      "learning_rate": 9.939949247384046e-05,
+      "loss": 0.49120208621025085,
+      "mean_token_accuracy": 0.8381295204162598,
+      "num_tokens": 21886.0,
+      "step": 26
+    },
+    {
+      "entropy": 0.3697093725204468,
+      "epoch": 0.10546875,
+      "grad_norm": 0.4351414144039154,
+      "learning_rate": 9.929548316723982e-05,
+      "loss": 0.4187180697917938,
+      "mean_token_accuracy": 0.8720238208770752,
+      "num_tokens": 22959.0,
+      "step": 27
+    },
+    {
+      "entropy": 0.5468494892120361,
+      "epoch": 0.109375,
+      "grad_norm": 0.7426009178161621,
+      "learning_rate": 9.918323459933005e-05,
+      "loss": 0.5927971601486206,
+      "mean_token_accuracy": 0.8241758346557617,
+      "num_tokens": 23648.0,
+      "step": 28
+    },
+    {
+      "entropy": 0.8542861342430115,
+      "epoch": 0.11328125,
+      "grad_norm": 1.2180156707763672,
+      "learning_rate": 9.906276553136923e-05,
+      "loss": 1.0376310348510742,
+      "mean_token_accuracy": 0.7510373592376709,
+      "num_tokens": 24228.0,
+      "step": 29
+    },
+    {
+      "entropy": 0.48639291524887085,
+      "epoch": 0.1171875,
+      "grad_norm": 0.5901576280593872,
+      "learning_rate": 9.893409609859222e-05,
+      "loss": 0.5231559872627258,
+      "mean_token_accuracy": 0.8418862819671631,
+      "num_tokens": 25005.0,
+      "step": 30
+    },
+    {
+      "entropy": 0.6086449027061462,
+      "epoch": 0.12109375,
+      "grad_norm": 0.5684905052185059,
+      "learning_rate": 9.879724780684519e-05,
+      "loss": 0.6100994944572449,
+      "mean_token_accuracy": 0.7975903749465942,
+      "num_tokens": 25945.0,
+      "step": 31
+    },
+    {
+      "entropy": 0.2810980975627899,
+      "epoch": 0.125,
+      "grad_norm": 0.8133203983306885,
+      "learning_rate": 9.865224352899119e-05,
+      "loss": 0.27467983961105347,
+      "mean_token_accuracy": 0.9043062329292297,
+      "num_tokens": 26524.0,
+      "step": 32
+    },
+    {
+      "entropy": 0.8997385501861572,
+      "epoch": 0.12890625,
+      "grad_norm": 0.7001259922981262,
+      "learning_rate": 9.849910750108717e-05,
+      "loss": 0.872958779335022,
+      "mean_token_accuracy": 0.75,
+      "num_tokens": 27339.0,
+      "step": 33
+    },
+    {
+      "entropy": 0.763770341873169,
+      "epoch": 0.1328125,
+      "grad_norm": 0.6955083012580872,
+      "learning_rate": 9.83378653183331e-05,
+      "loss": 0.8287037014961243,
+      "mean_token_accuracy": 0.7598253488540649,
+      "num_tokens": 28100.0,
+      "step": 34
+    },
+    {
+      "entropy": 0.3402330279350281,
+      "epoch": 0.13671875,
+      "grad_norm": 0.48411691188812256,
+      "learning_rate": 9.816854393079403e-05,
+      "loss": 0.36252474784851074,
+      "mean_token_accuracy": 0.8937259912490845,
+      "num_tokens": 28935.0,
+      "step": 35
+    },
+    {
+      "entropy": 0.29372096061706543,
+      "epoch": 0.140625,
+      "grad_norm": 0.6549316644668579,
+      "learning_rate": 9.799117163889559e-05,
+      "loss": 0.2963707745075226,
+      "mean_token_accuracy": 0.9077306985855103,
+      "num_tokens": 29714.0,
+      "step": 36
+    },
+    {
+      "entropy": 0.4241790473461151,
+      "epoch": 0.14453125,
+      "grad_norm": 0.7347589731216431,
+      "learning_rate": 9.780577808869398e-05,
+      "loss": 0.5015050172805786,
+      "mean_token_accuracy": 0.8459529876708984,
+      "num_tokens": 30523.0,
+      "step": 37
+    },
+    {
+      "entropy": 0.28753387928009033,
+      "epoch": 0.1484375,
+      "grad_norm": 0.719067394733429,
+      "learning_rate": 9.761239426692077e-05,
+      "loss": 0.2576276361942291,
+      "mean_token_accuracy": 0.9147540926933289,
+      "num_tokens": 31208.0,
+      "step": 38
+    },
+    {
+      "entropy": 0.475173681974411,
+      "epoch": 0.15234375,
+      "grad_norm": 1.1069060564041138,
+      "learning_rate": 9.741105249580383e-05,
+      "loss": 0.4583543539047241,
+      "mean_token_accuracy": 0.8554216623306274,
+      "num_tokens": 31761.0,
+      "step": 39
+    },
+    {
+      "entropy": 0.2746836841106415,
+      "epoch": 0.15625,
+      "grad_norm": 0.8300859332084656,
+      "learning_rate": 9.7201786427665e-05,
+      "loss": 0.2247270941734314,
+      "mean_token_accuracy": 0.9043062329292297,
+      "num_tokens": 32326.0,
+      "step": 40
+    },
+    {
+      "entropy": 0.5290616154670715,
+      "epoch": 0.16015625,
+      "grad_norm": 0.5069507956504822,
+      "learning_rate": 9.698463103929542e-05,
+      "loss": 0.4967139661312103,
+      "mean_token_accuracy": 0.8347305655479431,
+      "num_tokens": 33237.0,
+      "step": 41
+    },
+    {
+      "entropy": 0.17607364058494568,
+      "epoch": 0.1640625,
+      "grad_norm": 0.8130342364311218,
+      "learning_rate": 9.67596226261095e-05,
+      "loss": 0.18348810076713562,
+      "mean_token_accuracy": 0.9281437397003174,
+      "num_tokens": 33765.0,
+      "step": 42
+    },
+    {
+      "entropy": 0.9480698704719543,
+      "epoch": 0.16796875,
+      "grad_norm": 0.8172163963317871,
+      "learning_rate": 9.652679879607843e-05,
+      "loss": 0.9325052499771118,
+      "mean_token_accuracy": 0.7251114249229431,
+      "num_tokens": 34498.0,
+      "step": 43
+    },
+    {
+      "entropy": 0.4078804850578308,
+      "epoch": 0.171875,
+      "grad_norm": 0.7695475816726685,
+      "learning_rate": 9.628619846344454e-05,
+      "loss": 0.32897821068763733,
+      "mean_token_accuracy": 0.8977035284042358,
+      "num_tokens": 35059.0,
+      "step": 44
+    },
+    {
+      "entropy": 0.6567792296409607,
+      "epoch": 0.17578125,
+      "grad_norm": 0.6577677726745605,
+      "learning_rate": 9.603786184221693e-05,
+      "loss": 0.6604552268981934,
+      "mean_token_accuracy": 0.7876712083816528,
+      "num_tokens": 35718.0,
+      "step": 45
+    },
+    {
+      "entropy": 0.5705280900001526,
+      "epoch": 0.1796875,
+      "grad_norm": 0.5337263941764832,
+      "learning_rate": 9.57818304394503e-05,
+      "loss": 0.5775972604751587,
+      "mean_token_accuracy": 0.8275400996208191,
+      "num_tokens": 36508.0,
+      "step": 46
+    },
+    {
+      "entropy": 0.4061397612094879,
+      "epoch": 0.18359375,
+      "grad_norm": 0.6210362315177917,
+      "learning_rate": 9.551814704830734e-05,
+      "loss": 0.30573710799217224,
+      "mean_token_accuracy": 0.8992481231689453,
+      "num_tokens": 37606.0,
+      "step": 47
+    },
+    {
+      "entropy": 0.7427402138710022,
+      "epoch": 0.1875,
+      "grad_norm": 0.6562739610671997,
+      "learning_rate": 9.524685574090627e-05,
+      "loss": 0.8000867366790771,
+      "mean_token_accuracy": 0.7481698393821716,
+      "num_tokens": 38352.0,
+      "step": 48
+    },
+    {
+      "entropy": 0.5252540707588196,
+      "epoch": 0.19140625,
+      "grad_norm": 1.1183611154556274,
+      "learning_rate": 9.496800186095466e-05,
+      "loss": 0.5875065326690674,
+      "mean_token_accuracy": 0.8088235259056091,
+      "num_tokens": 38923.0,
+      "step": 49
+    },
+    {
+      "entropy": 0.15077699720859528,
+      "epoch": 0.1953125,
+      "grad_norm": 0.5114659070968628,
+      "learning_rate": 9.468163201617062e-05,
+      "loss": 0.1495133340358734,
+      "mean_token_accuracy": 0.9547738432884216,
+      "num_tokens": 39700.0,
+      "step": 50
+    },
+    {
+      "entropy": 0.5502139329910278,
+      "epoch": 0.19921875,
+      "grad_norm": 0.7144715785980225,
+      "learning_rate": 9.438779407049281e-05,
+      "loss": 0.5862892866134644,
+      "mean_token_accuracy": 0.8315789699554443,
+      "num_tokens": 40419.0,
+      "step": 51
+    },
+    {
+      "entropy": 0.48560580611228943,
+      "epoch": 0.203125,
+      "grad_norm": 0.513193666934967,
+      "learning_rate": 9.40865371360804e-05,
+      "loss": 0.47880712151527405,
+      "mean_token_accuracy": 0.8279411792755127,
+      "num_tokens": 41202.0,
+      "step": 52
+    },
+    {
+      "entropy": 0.33771631121635437,
+      "epoch": 0.20703125,
+      "grad_norm": 0.5135825872421265,
+      "learning_rate": 9.377791156510455e-05,
+      "loss": 0.28588980436325073,
+      "mean_token_accuracy": 0.9021447896957397,
+      "num_tokens": 42455.0,
+      "step": 53
+    },
+    {
+      "entropy": 0.6165871024131775,
+      "epoch": 0.2109375,
+      "grad_norm": 0.48555585741996765,
+      "learning_rate": 9.346196894133239e-05,
+      "loss": 0.6004568338394165,
+      "mean_token_accuracy": 0.8091286420822144,
+      "num_tokens": 43493.0,
+      "step": 54
+    },
+    {
+      "entropy": 0.7081799507141113,
+      "epoch": 0.21484375,
+      "grad_norm": 1.0581609010696411,
+      "learning_rate": 9.313876207150543e-05,
+      "loss": 0.8294097185134888,
+      "mean_token_accuracy": 0.78925621509552,
+      "num_tokens": 44125.0,
+      "step": 55
+    },
+    {
+      "entropy": 0.45999354124069214,
+      "epoch": 0.21875,
+      "grad_norm": 0.4506646394729614,
+      "learning_rate": 9.280834497651334e-05,
+      "loss": 0.5012324452400208,
+      "mean_token_accuracy": 0.8410757780075073,
+      "num_tokens": 44991.0,
+      "step": 56
+    },
+    {
+      "entropy": 0.5937639474868774,
+      "epoch": 0.22265625,
+      "grad_norm": 0.54532790184021,
+      "learning_rate": 9.247077288236488e-05,
+      "loss": 0.6704734563827515,
+      "mean_token_accuracy": 0.8050089478492737,
+      "num_tokens": 45861.0,
+      "step": 57
+    },
+    {
+      "entropy": 0.6354159712791443,
+      "epoch": 0.2265625,
+      "grad_norm": 0.4860483705997467,
+      "learning_rate": 9.212610221095748e-05,
+      "loss": 0.6274347305297852,
+      "mean_token_accuracy": 0.8134490251541138,
+      "num_tokens": 46838.0,
+      "step": 58
+    },
+    {
+      "entropy": 0.5907365679740906,
+      "epoch": 0.23046875,
+      "grad_norm": 0.6250556707382202,
+      "learning_rate": 9.177439057064683e-05,
+      "loss": 0.6647694706916809,
+      "mean_token_accuracy": 0.7956834435462952,
+      "num_tokens": 47622.0,
+      "step": 59
+    },
+    {
+      "entropy": 0.49545010924339294,
+      "epoch": 0.234375,
+      "grad_norm": 0.50145423412323,
+      "learning_rate": 9.141569674661817e-05,
+      "loss": 0.5231200456619263,
+      "mean_token_accuracy": 0.8244473338127136,
+      "num_tokens": 48454.0,
+      "step": 60
+    },
+    {
+      "entropy": 0.6162831783294678,
+      "epoch": 0.23828125,
+      "grad_norm": 0.4918208420276642,
+      "learning_rate": 9.105008069106093e-05,
+      "loss": 0.6733858585357666,
+      "mean_token_accuracy": 0.8091286420822144,
+      "num_tokens": 49253.0,
+      "step": 61
+    },
+    {
+      "entropy": 0.05496763810515404,
+      "epoch": 0.2421875,
+      "grad_norm": 1.3541446924209595,
+      "learning_rate": 9.067760351314838e-05,
+      "loss": 0.10028333216905594,
+      "mean_token_accuracy": 0.9659090638160706,
+      "num_tokens": 49514.0,
+      "step": 62
+    },
+    {
+      "entropy": 0.5858029723167419,
+      "epoch": 0.24609375,
+      "grad_norm": 0.41623160243034363,
+      "learning_rate": 9.029832746882371e-05,
+      "loss": 0.6132221221923828,
+      "mean_token_accuracy": 0.820853054523468,
+      "num_tokens": 50637.0,
+      "step": 63
+    },
+    {
+      "entropy": 0.11868977546691895,
+      "epoch": 0.25,
+      "grad_norm": 0.5636643171310425,
+      "learning_rate": 8.991231595039465e-05,
+      "loss": 0.0864829421043396,
+      "mean_token_accuracy": 0.9688888788223267,
+      "num_tokens": 51241.0,
+      "step": 64
+    },
+    {
+      "entropy": 0.5117590427398682,
+      "epoch": 0.25390625,
+      "grad_norm": 0.6280642747879028,
+      "learning_rate": 8.951963347593797e-05,
+      "loss": 0.38439318537712097,
+      "mean_token_accuracy": 0.8670520186424255,
+      "num_tokens": 52015.0,
+      "step": 65
+    },
+    {
+      "entropy": 0.5181878209114075,
+      "epoch": 0.2578125,
+      "grad_norm": 0.9919320940971375,
+      "learning_rate": 8.912034567851599e-05,
+      "loss": 0.5401250720024109,
+      "mean_token_accuracy": 0.7982062697410583,
+      "num_tokens": 52616.0,
+      "step": 66
+    },
+    {
+      "entropy": 0.23600083589553833,
+      "epoch": 0.26171875,
+      "grad_norm": 0.6863732933998108,
+      "learning_rate": 8.871451929520663e-05,
+      "loss": 0.25889232754707336,
+      "mean_token_accuracy": 0.9220778942108154,
+      "num_tokens": 53327.0,
+      "step": 67
+    },
+    {
+      "entropy": 0.47514960169792175,
+      "epoch": 0.265625,
+      "grad_norm": 0.708480715751648,
+      "learning_rate": 8.83022221559489e-05,
+      "loss": 0.46035659313201904,
+      "mean_token_accuracy": 0.8456140160560608,
+      "num_tokens": 54106.0,
+      "step": 68
+    },
+    {
+      "entropy": 0.1594257950782776,
+      "epoch": 0.26953125,
+      "grad_norm": 0.7160263061523438,
+      "learning_rate": 8.78835231722059e-05,
+      "loss": 0.15633094310760498,
+      "mean_token_accuracy": 0.9552238583564758,
+      "num_tokens": 54663.0,
+      "step": 69
+    },
+    {
+      "entropy": 0.3020382523536682,
+      "epoch": 0.2734375,
+      "grad_norm": 0.6970436573028564,
+      "learning_rate": 8.745849232544681e-05,
+      "loss": 0.29399704933166504,
+      "mean_token_accuracy": 0.882539689540863,
+      "num_tokens": 55493.0,
+      "step": 70
+    },
+    {
+      "entropy": 0.5089768767356873,
+      "epoch": 0.27734375,
+      "grad_norm": 0.3825555145740509,
+      "learning_rate": 8.702720065545024e-05,
+      "loss": 0.4680228531360626,
+      "mean_token_accuracy": 0.8552875518798828,
+      "num_tokens": 56629.0,
+      "step": 71
+    },
+    {
+      "entropy": 1.1572331190109253,
+      "epoch": 0.28125,
+      "grad_norm": 0.7397168874740601,
+      "learning_rate": 8.658972024843062e-05,
+      "loss": 1.1789411306381226,
+      "mean_token_accuracy": 0.6794354915618896,
+      "num_tokens": 57563.0,
+      "step": 72
+    },
+    {
+      "entropy": 0.32222115993499756,
+      "epoch": 0.28515625,
+      "grad_norm": 0.5344856977462769,
+      "learning_rate": 8.614612422498964e-05,
+      "loss": 0.3255745768547058,
+      "mean_token_accuracy": 0.8844155669212341,
+      "num_tokens": 58670.0,
+      "step": 73
+    },
+    {
+      "entropy": 0.4632827937602997,
+      "epoch": 0.2890625,
+      "grad_norm": 0.5455396771430969,
+      "learning_rate": 8.569648672789497e-05,
+      "loss": 0.4700681269168854,
+      "mean_token_accuracy": 0.840228259563446,
+      "num_tokens": 59428.0,
+      "step": 74
+    },
+    {
+      "entropy": 0.1857202798128128,
+      "epoch": 0.29296875,
+      "grad_norm": 0.5769023895263672,
+      "learning_rate": 8.524088290968781e-05,
+      "loss": 0.19467991590499878,
+      "mean_token_accuracy": 0.9217687249183655,
+      "num_tokens": 60095.0,
+      "step": 75
+    },
+    {
+      "entropy": 0.3141404986381531,
+      "epoch": 0.296875,
+      "grad_norm": 0.29498976469039917,
+      "learning_rate": 8.47793889201221e-05,
+      "loss": 0.31678324937820435,
+      "mean_token_accuracy": 0.9121830463409424,
+      "num_tokens": 61765.0,
+      "step": 76
+    },
+    {
+      "entropy": 0.2404460608959198,
+      "epoch": 0.30078125,
+      "grad_norm": 0.4017998278141022,
+      "learning_rate": 8.43120818934367e-05,
+      "loss": 0.23486091196537018,
+      "mean_token_accuracy": 0.9183841943740845,
+      "num_tokens": 63570.0,
+      "step": 77
+    },
+    {
+      "entropy": 0.5518016219139099,
+      "epoch": 0.3046875,
+      "grad_norm": 0.49208730459213257,
+      "learning_rate": 8.383903993546311e-05,
+      "loss": 0.6085214018821716,
+      "mean_token_accuracy": 0.8143872022628784,
+      "num_tokens": 64765.0,
+      "step": 78
+    },
+    {
+      "entropy": 0.18448811769485474,
+      "epoch": 0.30859375,
+      "grad_norm": 0.37179243564605713,
+      "learning_rate": 8.336034211057098e-05,
+      "loss": 0.14006929099559784,
+      "mean_token_accuracy": 0.9479553699493408,
+      "num_tokens": 65685.0,
+      "step": 79
+    },
+    {
+      "entropy": 0.42918166518211365,
+      "epoch": 0.3125,
+      "grad_norm": 0.442133367061615,
+      "learning_rate": 8.28760684284532e-05,
+      "loss": 0.4223015606403351,
+      "mean_token_accuracy": 0.8700494766235352,
+      "num_tokens": 66558.0,
+      "step": 80
+    },
+    {
+      "entropy": 0.5193983912467957,
+      "epoch": 0.31640625,
+      "grad_norm": 0.7973830103874207,
+      "learning_rate": 8.238629983075294e-05,
+      "loss": 0.5227593183517456,
+      "mean_token_accuracy": 0.8419753313064575,
+      "num_tokens": 67021.0,
+      "step": 81
+    },
+    {
+      "entropy": 0.6162667274475098,
+      "epoch": 0.3203125,
+      "grad_norm": 0.3803671896457672,
+      "learning_rate": 8.18911181775353e-05,
+      "loss": 0.5831434726715088,
+      "mean_token_accuracy": 0.8372319936752319,
+      "num_tokens": 68815.0,
+      "step": 82
+    },
+    {
+      "entropy": 0.5226625800132751,
+      "epoch": 0.32421875,
+      "grad_norm": 0.6740264892578125,
+      "learning_rate": 8.139060623360493e-05,
+      "loss": 0.5770704746246338,
+      "mean_token_accuracy": 0.8193717002868652,
+      "num_tokens": 69250.0,
+      "step": 83
+    },
+    {
+      "entropy": 0.2120583951473236,
+      "epoch": 0.328125,
+      "grad_norm": 0.3773080110549927,
+      "learning_rate": 8.088484765467286e-05,
+      "loss": 0.17072270810604095,
+      "mean_token_accuracy": 0.9328436255455017,
+      "num_tokens": 70269.0,
+      "step": 84
+    },
+    {
+      "entropy": 0.5727735757827759,
+      "epoch": 0.33203125,
+      "grad_norm": 0.47534501552581787,
+      "learning_rate": 8.037392697337418e-05,
+      "loss": 0.6191591024398804,
+      "mean_token_accuracy": 0.8163265585899353,
+      "num_tokens": 71812.0,
+      "step": 85
+    },
+    {
+      "entropy": 0.5548444390296936,
+      "epoch": 0.3359375,
+      "grad_norm": 0.33695611357688904,
+      "learning_rate": 7.985792958513931e-05,
+      "loss": 0.6584370136260986,
+      "mean_token_accuracy": 0.8497409224510193,
+      "num_tokens": 73860.0,
+      "step": 86
+    },
+    {
+      "entropy": 0.5205814242362976,
+      "epoch": 0.33984375,
+      "grad_norm": 1.0107587575912476,
+      "learning_rate": 7.93369417339209e-05,
+      "loss": 0.5680365562438965,
+      "mean_token_accuracy": 0.8446969985961914,
+      "num_tokens": 74494.0,
+      "step": 87
+    },
+    {
+      "entropy": 0.480228990316391,
+      "epoch": 0.34375,
+      "grad_norm": 0.604826807975769,
+      "learning_rate": 7.881105049777901e-05,
+      "loss": 0.428594172000885,
+      "mean_token_accuracy": 0.8599476218223572,
+      "num_tokens": 75328.0,
+      "step": 88
+    },
+    {
+      "entropy": 0.5686553120613098,
+      "epoch": 0.34765625,
+      "grad_norm": 0.5422868728637695,
+      "learning_rate": 7.828034377432693e-05,
+      "loss": 0.5080078840255737,
+      "mean_token_accuracy": 0.8353344798088074,
+      "num_tokens": 75984.0,
+      "step": 89
+    },
+    {
+      "entropy": 0.41963568329811096,
+      "epoch": 0.3515625,
+      "grad_norm": 0.3786294460296631,
+      "learning_rate": 7.774491026603985e-05,
+      "loss": 0.3882997930049896,
+      "mean_token_accuracy": 0.8694127798080444,
+      "num_tokens": 77201.0,
+      "step": 90
+    },
+    {
+      "entropy": 0.8213188648223877,
+      "epoch": 0.35546875,
+      "grad_norm": 0.5294942855834961,
+      "learning_rate": 7.720483946542914e-05,
+      "loss": 0.8224100470542908,
+      "mean_token_accuracy": 0.76325523853302,
+      "num_tokens": 78072.0,
+      "step": 91
+    },
+    {
+      "entropy": 0.35614386200904846,
+      "epoch": 0.359375,
+      "grad_norm": 0.5392538905143738,
+      "learning_rate": 7.666022164008457e-05,
+      "loss": 0.2930217683315277,
+      "mean_token_accuracy": 0.8999999761581421,
+      "num_tokens": 79347.0,
+      "step": 92
+    },
+    {
+      "entropy": 0.38908302783966064,
+      "epoch": 0.36328125,
+      "grad_norm": 0.7368007302284241,
+      "learning_rate": 7.611114781758692e-05,
+      "loss": 0.4357201159000397,
+      "mean_token_accuracy": 0.8640483617782593,
+      "num_tokens": 80066.0,
+      "step": 93
+    },
+    {
+      "entropy": 0.4956449568271637,
+      "epoch": 0.3671875,
+      "grad_norm": 0.5707206130027771,
+      "learning_rate": 7.555770977029367e-05,
+      "loss": 0.5065670013427734,
+      "mean_token_accuracy": 0.8519313335418701,
+      "num_tokens": 80697.0,
+      "step": 94
+    },
+    {
+      "entropy": 0.5943759083747864,
+      "epoch": 0.37109375,
+      "grad_norm": 0.7254245281219482,
+      "learning_rate": 7.500000000000001e-05,
+      "loss": 0.6552464365959167,
+      "mean_token_accuracy": 0.7767584323883057,
+      "num_tokens": 81336.0,
+      "step": 95
+    },
+    {
+      "entropy": 0.532646894454956,
+      "epoch": 0.375,
+      "grad_norm": 0.5899369120597839,
+      "learning_rate": 7.443811172247821e-05,
+      "loss": 0.4484238624572754,
+      "mean_token_accuracy": 0.8579465746879578,
+      "num_tokens": 82184.0,
+      "step": 96
+    },
+    {
+      "entropy": 0.5296929478645325,
+      "epoch": 0.37890625,
+      "grad_norm": 0.5489293336868286,
+      "learning_rate": 7.387213885189746e-05,
+      "loss": 0.5374542474746704,
+      "mean_token_accuracy": 0.8023715615272522,
+      "num_tokens": 82759.0,
+      "step": 97
+    },
+    {
+      "entropy": 0.1481165885925293,
+      "epoch": 0.3828125,
+      "grad_norm": 0.31081852316856384,
+      "learning_rate": 7.330217598512695e-05,
+      "loss": 0.14701320230960846,
+      "mean_token_accuracy": 0.9493827223777771,
+      "num_tokens": 83934.0,
+      "step": 98
+    },
+    {
+      "entropy": 0.9935259222984314,
+      "epoch": 0.38671875,
+      "grad_norm": 0.6228557229042053,
+      "learning_rate": 7.272831838592503e-05,
+      "loss": 1.0415065288543701,
+      "mean_token_accuracy": 0.7242603302001953,
+      "num_tokens": 84835.0,
+      "step": 99
+    },
+    {
+      "entropy": 0.24420775473117828,
+      "epoch": 0.390625,
+      "grad_norm": 0.38697612285614014,
+      "learning_rate": 7.215066196901676e-05,
+      "loss": 0.2529013454914093,
+      "mean_token_accuracy": 0.9107373952865601,
+      "num_tokens": 86058.0,
+      "step": 100
+    },
+    {
+      "entropy": 0.2862064242362976,
+      "epoch": 0.39453125,
+      "grad_norm": 0.4424213767051697,
+      "learning_rate": 7.156930328406268e-05,
+      "loss": 0.23783738911151886,
+      "mean_token_accuracy": 0.9170854091644287,
+      "num_tokens": 86927.0,
+      "step": 101
+    },
+    {
+      "entropy": 0.2807987332344055,
+      "epoch": 0.3984375,
+      "grad_norm": 0.770887553691864,
+      "learning_rate": 7.098433949952146e-05,
+      "loss": 0.3131881356239319,
+      "mean_token_accuracy": 0.8866396546363831,
+      "num_tokens": 87579.0,
+      "step": 102
+    },
+    {
+      "entropy": 0.41032883524894714,
+      "epoch": 0.40234375,
+      "grad_norm": 0.5959399938583374,
+      "learning_rate": 7.039586838640919e-05,
+      "loss": 0.46824127435684204,
+      "mean_token_accuracy": 0.8435373902320862,
+      "num_tokens": 88376.0,
+      "step": 103
+    },
+    {
+      "entropy": 0.32393795251846313,
+      "epoch": 0.40625,
+      "grad_norm": 0.5406798720359802,
+      "learning_rate": 6.980398830195785e-05,
+      "loss": 0.34825822710990906,
+      "mean_token_accuracy": 0.8803301453590393,
+      "num_tokens": 89642.0,
+      "step": 104
+    },
+    {
+      "entropy": 0.47948575019836426,
+      "epoch": 0.41015625,
+      "grad_norm": 0.5012584924697876,
+      "learning_rate": 6.920879817317589e-05,
+      "loss": 0.5121287107467651,
+      "mean_token_accuracy": 0.8364388942718506,
+      "num_tokens": 90687.0,
+      "step": 105
+    },
+    {
+      "entropy": 0.22555242478847504,
+      "epoch": 0.4140625,
+      "grad_norm": 0.4911995232105255,
+      "learning_rate": 6.861039748031351e-05,
+      "loss": 0.2424127608537674,
+      "mean_token_accuracy": 0.9094339609146118,
+      "num_tokens": 91785.0,
+      "step": 106
+    },
+    {
+      "entropy": 0.6763952970504761,
+      "epoch": 0.41796875,
+      "grad_norm": 0.75534987449646,
+      "learning_rate": 6.800888624023553e-05,
+      "loss": 0.6895910501480103,
+      "mean_token_accuracy": 0.7877984046936035,
+      "num_tokens": 92534.0,
+      "step": 107
+    },
+    {
+      "entropy": 0.2540339231491089,
+      "epoch": 0.421875,
+      "grad_norm": 0.4766096770763397,
+      "learning_rate": 6.740436498970452e-05,
+      "loss": 0.24226245284080505,
+      "mean_token_accuracy": 0.9155405163764954,
+      "num_tokens": 93241.0,
+      "step": 108
+    },
+    {
+      "entropy": 0.3385879397392273,
+      "epoch": 0.42578125,
+      "grad_norm": 0.5595477819442749,
+      "learning_rate": 6.679693476857711e-05,
+      "loss": 0.2845480144023895,
+      "mean_token_accuracy": 0.8961864113807678,
+      "num_tokens": 93793.0,
+      "step": 109
+    },
+    {
+      "entropy": 0.49887916445732117,
+      "epoch": 0.4296875,
+      "grad_norm": 0.5290985703468323,
+      "learning_rate": 6.618669710291606e-05,
+      "loss": 0.5168520212173462,
+      "mean_token_accuracy": 0.8165869116783142,
+      "num_tokens": 94471.0,
+      "step": 110
+    },
+    {
+      "entropy": 0.38629135489463806,
+      "epoch": 0.43359375,
+      "grad_norm": 0.36983832716941833,
+      "learning_rate": 6.557375398802123e-05,
+      "loss": 0.40279698371887207,
+      "mean_token_accuracy": 0.8754246830940247,
+      "num_tokens": 95441.0,
+      "step": 111
+    },
+    {
+      "entropy": 0.5321238040924072,
+      "epoch": 0.4375,
+      "grad_norm": 0.39806121587753296,
+      "learning_rate": 6.495820787138209e-05,
+      "loss": 0.5178214907646179,
+      "mean_token_accuracy": 0.8385502696037292,
+      "num_tokens": 96707.0,
+      "step": 112
+    },
+    {
+      "entropy": 0.3818260133266449,
+      "epoch": 0.44140625,
+      "grad_norm": 0.525242805480957,
+      "learning_rate": 6.434016163555452e-05,
+      "loss": 0.3843124806880951,
+      "mean_token_accuracy": 0.8673341870307922,
+      "num_tokens": 97715.0,
+      "step": 113
+    },
+    {
+      "entropy": 0.28144916892051697,
+      "epoch": 0.4453125,
+      "grad_norm": 0.5517542958259583,
+      "learning_rate": 6.371971858096508e-05,
+      "loss": 0.23217296600341797,
+      "mean_token_accuracy": 0.9144737124443054,
+      "num_tokens": 98726.0,
+      "step": 114
+    },
+    {
+      "entropy": 0.5824229121208191,
+      "epoch": 0.44921875,
+      "grad_norm": 0.5402820706367493,
+      "learning_rate": 6.30969824086453e-05,
+      "loss": 0.5959458947181702,
+      "mean_token_accuracy": 0.8171700835227966,
+      "num_tokens": 99424.0,
+      "step": 115
+    },
+    {
+      "entropy": 0.8030366897583008,
+      "epoch": 0.453125,
+      "grad_norm": 0.5737054347991943,
+      "learning_rate": 6.247205720289907e-05,
+      "loss": 0.8519771099090576,
+      "mean_token_accuracy": 0.7758865356445312,
+      "num_tokens": 100180.0,
+      "step": 116
+    },
+    {
+      "entropy": 0.7312548756599426,
+      "epoch": 0.45703125,
+      "grad_norm": 0.5220651626586914,
+      "learning_rate": 6.184504741390596e-05,
+      "loss": 0.7618618607521057,
+      "mean_token_accuracy": 0.7858064770698547,
+      "num_tokens": 101010.0,
+      "step": 117
+    },
+    {
+      "entropy": 0.7373753190040588,
+      "epoch": 0.4609375,
+      "grad_norm": 0.6606554985046387,
+      "learning_rate": 6.121605784026339e-05,
+      "loss": 0.8617627620697021,
+      "mean_token_accuracy": 0.782608687877655,
+      "num_tokens": 101582.0,
+      "step": 118
+    },
+    {
+      "entropy": 0.41949662566185,
+      "epoch": 0.46484375,
+      "grad_norm": 0.7282864451408386,
+      "learning_rate": 6.058519361147055e-05,
+      "loss": 0.3674086630344391,
+      "mean_token_accuracy": 0.8489583134651184,
+      "num_tokens": 102164.0,
+      "step": 119
+    },
+    {
+      "entropy": 0.5909773707389832,
+      "epoch": 0.46875,
+      "grad_norm": 0.6897526979446411,
+      "learning_rate": 5.995256017035703e-05,
+      "loss": 0.597702145576477,
+      "mean_token_accuracy": 0.8135592937469482,
+      "num_tokens": 102773.0,
+      "step": 120
+    },
+    {
+      "entropy": 0.3142830431461334,
+      "epoch": 0.47265625,
+      "grad_norm": 0.44246742129325867,
+      "learning_rate": 5.9318263255459116e-05,
+      "loss": 0.33385440707206726,
+      "mean_token_accuracy": 0.9006896615028381,
+      "num_tokens": 104003.0,
+      "step": 121
+    },
+    {
+      "entropy": 0.6774213314056396,
+      "epoch": 0.4765625,
+      "grad_norm": 0.5509549379348755,
+      "learning_rate": 5.868240888334653e-05,
+      "loss": 0.7487766742706299,
+      "mean_token_accuracy": 0.7820324301719666,
+      "num_tokens": 104800.0,
+      "step": 122
+    },
+    {
+      "entropy": 0.7054826021194458,
+      "epoch": 0.48046875,
+      "grad_norm": 0.5595342516899109,
+      "learning_rate": 5.804510333090287e-05,
+      "loss": 0.6620463132858276,
+      "mean_token_accuracy": 0.7893333435058594,
+      "num_tokens": 105601.0,
+      "step": 123
+    },
+    {
+      "entropy": 0.551423192024231,
+      "epoch": 0.484375,
+      "grad_norm": 1.1154266595840454,
+      "learning_rate": 5.740645311756245e-05,
+      "loss": 0.4736259877681732,
+      "mean_token_accuracy": 0.8301886916160583,
+      "num_tokens": 106123.0,
+      "step": 124
+    },
+    {
+      "entropy": 0.5324803590774536,
+      "epoch": 0.48828125,
+      "grad_norm": 0.5987504720687866,
+      "learning_rate": 5.6766564987506566e-05,
+      "loss": 0.4106941819190979,
+      "mean_token_accuracy": 0.8658718466758728,
+      "num_tokens": 106937.0,
+      "step": 125
+    },
+    {
+      "entropy": 0.24800963699817657,
+      "epoch": 0.4921875,
+      "grad_norm": 0.56256502866745,
+      "learning_rate": 5.6125545891822274e-05,
+      "loss": 0.24778658151626587,
+      "mean_token_accuracy": 0.8974999785423279,
+      "num_tokens": 107471.0,
+      "step": 126
+    },
+    {
+      "entropy": 0.7553551197052002,
+      "epoch": 0.49609375,
+      "grad_norm": 0.6148009896278381,
+      "learning_rate": 5.548350297062659e-05,
+      "loss": 0.8133091926574707,
+      "mean_token_accuracy": 0.7936508059501648,
+      "num_tokens": 108104.0,
+      "step": 127
+    },
+    {
+      "entropy": 0.4535163938999176,
+      "epoch": 0.5,
+      "grad_norm": 0.5074506402015686,
+      "learning_rate": 5.484054353515896e-05,
+      "loss": 0.47288593649864197,
+      "mean_token_accuracy": 0.8372781276702881,
+      "num_tokens": 108859.0,
+      "step": 128
+    },
+    {
+      "entropy": 0.26084503531455994,
+      "epoch": 0.50390625,
+      "grad_norm": 0.4284311532974243,
+      "learning_rate": 5.419677504984534e-05,
+      "loss": 0.3021049499511719,
+      "mean_token_accuracy": 0.9168975353240967,
+      "num_tokens": 110907.0,
+      "step": 129
+    },
+    {
+      "entropy": 0.6347863674163818,
+      "epoch": 0.5078125,
+      "grad_norm": 0.48329687118530273,
+      "learning_rate": 5.355230511433651e-05,
+      "loss": 0.6871066093444824,
+      "mean_token_accuracy": 0.8103864789009094,
+      "num_tokens": 111809.0,
+      "step": 130
+    },
+    {
+      "entropy": 0.6220738887786865,
+      "epoch": 0.51171875,
+      "grad_norm": 0.4319831430912018,
+      "learning_rate": 5.290724144552379e-05,
+      "loss": 0.5937281250953674,
+      "mean_token_accuracy": 0.8135765194892883,
+      "num_tokens": 112871.0,
+      "step": 131
+    },
+    {
+      "entropy": 0.6094347238540649,
+      "epoch": 0.515625,
+      "grad_norm": 0.42872029542922974,
+      "learning_rate": 5.226169185953532e-05,
+      "loss": 0.6027806401252747,
+      "mean_token_accuracy": 0.8122222423553467,
+      "num_tokens": 113833.0,
+      "step": 132
+    },
+    {
+      "entropy": 0.14849169552326202,
+      "epoch": 0.51953125,
+      "grad_norm": 0.5918900370597839,
+      "learning_rate": 5.1615764253715536e-05,
+      "loss": 0.12953366339206696,
+      "mean_token_accuracy": 0.9523809552192688,
+      "num_tokens": 114386.0,
+      "step": 133
+    },
+    {
+      "entropy": 0.7135059237480164,
+      "epoch": 0.5234375,
+      "grad_norm": 0.48909538984298706,
+      "learning_rate": 5.096956658859122e-05,
+      "loss": 0.7000653743743896,
+      "mean_token_accuracy": 0.7899761199951172,
+      "num_tokens": 115283.0,
+      "step": 134
+    },
+    {
+      "entropy": 0.2778969407081604,
+      "epoch": 0.52734375,
+      "grad_norm": 0.6036175489425659,
+      "learning_rate": 5.0323206869826966e-05,
+      "loss": 0.26437070965766907,
+      "mean_token_accuracy": 0.9032257795333862,
+      "num_tokens": 115898.0,
+      "step": 135
+    },
+    {
+      "entropy": 0.4649325907230377,
+      "epoch": 0.53125,
+      "grad_norm": 0.4481104910373688,
+      "learning_rate": 4.967679313017303e-05,
+      "loss": 0.46719998121261597,
+      "mean_token_accuracy": 0.8412017226219177,
+      "num_tokens": 116657.0,
+      "step": 136
+    },
+    {
+      "entropy": 0.547878623008728,
+      "epoch": 0.53515625,
+      "grad_norm": 0.7016973495483398,
+      "learning_rate": 4.903043341140879e-05,
+      "loss": 0.48092278838157654,
+      "mean_token_accuracy": 0.864628791809082,
+      "num_tokens": 117162.0,
+      "step": 137
+    },
+    {
+      "entropy": 0.5097542405128479,
+      "epoch": 0.5390625,
+      "grad_norm": 0.4204660952091217,
+      "learning_rate": 4.8384235746284476e-05,
+      "loss": 0.5161705613136292,
+      "mean_token_accuracy": 0.8362168669700623,
+      "num_tokens": 118078.0,
+      "step": 138
+    },
+    {
+      "entropy": 0.557927668094635,
+      "epoch": 0.54296875,
+      "grad_norm": 0.9529353976249695,
+      "learning_rate": 4.7738308140464685e-05,
+      "loss": 0.5611956119537354,
+      "mean_token_accuracy": 0.8146551847457886,
+      "num_tokens": 118671.0,
+      "step": 139
+    },
+    {
+      "entropy": 0.2800652086734772,
+      "epoch": 0.546875,
+      "grad_norm": 0.5067075490951538,
+      "learning_rate": 4.709275855447621e-05,
+      "loss": 0.24611259996891022,
+      "mean_token_accuracy": 0.9192463159561157,
+      "num_tokens": 119790.0,
+      "step": 140
+    },
+    {
+      "entropy": 0.5083785057067871,
+      "epoch": 0.55078125,
+      "grad_norm": 0.4677392244338989,
+      "learning_rate": 4.6447694885663514e-05,
+      "loss": 0.5404981374740601,
+      "mean_token_accuracy": 0.8203017711639404,
+      "num_tokens": 120579.0,
+      "step": 141
+    },
+    {
+      "entropy": 0.6280187964439392,
+      "epoch": 0.5546875,
+      "grad_norm": 1.1353791952133179,
+      "learning_rate": 4.5803224950154656e-05,
+      "loss": 0.6394505500793457,
+      "mean_token_accuracy": 0.7982345819473267,
+      "num_tokens": 121428.0,
+      "step": 142
+    },
+    {
+      "entropy": 0.6738426685333252,
+      "epoch": 0.55859375,
+      "grad_norm": 0.5054082870483398,
+      "learning_rate": 4.515945646484105e-05,
+      "loss": 0.6168086528778076,
+      "mean_token_accuracy": 0.80580073595047,
+      "num_tokens": 122302.0,
+      "step": 143
+    },
+    {
+      "entropy": 0.4337348937988281,
+      "epoch": 0.5625,
+      "grad_norm": 0.41570982336997986,
+      "learning_rate": 4.451649702937342e-05,
+      "loss": 0.4383181929588318,
+      "mean_token_accuracy": 0.8503937125205994,
+      "num_tokens": 123143.0,
+      "step": 144
+    },
+    {
+      "entropy": 0.678224503993988,
+      "epoch": 0.56640625,
+      "grad_norm": 0.5669712424278259,
+      "learning_rate": 4.387445410817774e-05,
+      "loss": 0.7309865951538086,
+      "mean_token_accuracy": 0.7887538075447083,
+      "num_tokens": 123869.0,
+      "step": 145
+    },
+    {
+      "entropy": 0.3064025640487671,
+      "epoch": 0.5703125,
+      "grad_norm": 0.6317686438560486,
+      "learning_rate": 4.323343501249346e-05,
+      "loss": 0.29763591289520264,
+      "mean_token_accuracy": 0.8975903391838074,
+      "num_tokens": 124687.0,
+      "step": 146
+    },
+    {
+      "entropy": 0.6311812400817871,
+      "epoch": 0.57421875,
+      "grad_norm": 1.0175809860229492,
+      "learning_rate": 4.259354688243757e-05,
+      "loss": 0.5514286160469055,
+      "mean_token_accuracy": 0.8260869383811951,
+      "num_tokens": 125323.0,
+      "step": 147
+    },
+    {
+      "entropy": 0.3044798970222473,
+      "epoch": 0.578125,
+      "grad_norm": 0.4535091519355774,
+      "learning_rate": 4.195489666909713e-05,
+      "loss": 0.2599211037158966,
+      "mean_token_accuracy": 0.9069182276725769,
+      "num_tokens": 126558.0,
+      "step": 148
+    },
+    {
+      "entropy": 0.601833164691925,
+      "epoch": 0.58203125,
+      "grad_norm": 0.5593596696853638,
+      "learning_rate": 4.131759111665349e-05,
+      "loss": 0.6810619235038757,
+      "mean_token_accuracy": 0.817251443862915,
+      "num_tokens": 127345.0,
+      "step": 149
+    },
+    {
+      "entropy": 0.28089091181755066,
+      "epoch": 0.5859375,
+      "grad_norm": 0.34061604738235474,
+      "learning_rate": 4.06817367445409e-05,
+      "loss": 0.2891358733177185,
+      "mean_token_accuracy": 0.9154518842697144,
+      "num_tokens": 128877.0,
+      "step": 150
+    },
+    {
+      "entropy": 0.24683843553066254,
+      "epoch": 0.58984375,
+      "grad_norm": 0.4543762803077698,
+      "learning_rate": 4.004743982964298e-05,
+      "loss": 0.2402358502149582,
+      "mean_token_accuracy": 0.9221938848495483,
+      "num_tokens": 130148.0,
+      "step": 151
+    },
+    {
+      "entropy": 0.5143784880638123,
+      "epoch": 0.59375,
+      "grad_norm": 0.411024272441864,
+      "learning_rate": 3.941480638852948e-05,
+      "loss": 0.47098883986473083,
+      "mean_token_accuracy": 0.8436781764030457,
+      "num_tokens": 131116.0,
+      "step": 152
+    },
+    {
+      "entropy": 0.34330156445503235,
+      "epoch": 0.59765625,
+      "grad_norm": 0.5047693252563477,
+      "learning_rate": 3.878394215973663e-05,
+      "loss": 0.2646324038505554,
+      "mean_token_accuracy": 0.9056603908538818,
+      "num_tokens": 131891.0,
+      "step": 153
+    },
+    {
+      "entropy": 1.0674253702163696,
+      "epoch": 0.6015625,
+      "grad_norm": 0.8713217973709106,
+      "learning_rate": 3.815495258609404e-05,
+      "loss": 1.192018985748291,
+      "mean_token_accuracy": 0.6854599118232727,
+      "num_tokens": 132710.0,
+      "step": 154
+    },
+    {
+      "entropy": 0.32038262486457825,
+      "epoch": 0.60546875,
+      "grad_norm": 0.7330225706100464,
+      "learning_rate": 3.752794279710094e-05,
+      "loss": 0.28299516439437866,
+      "mean_token_accuracy": 0.9047619104385376,
+      "num_tokens": 133397.0,
+      "step": 155
+    },
+    {
+      "entropy": 0.2789649963378906,
+      "epoch": 0.609375,
+      "grad_norm": 0.5472933053970337,
+      "learning_rate": 3.6903017591354706e-05,
+      "loss": 0.27893009781837463,
+      "mean_token_accuracy": 0.8918032646179199,
+      "num_tokens": 134585.0,
+      "step": 156
+    },
+    {
+      "entropy": 0.2711874544620514,
+      "epoch": 0.61328125,
+      "grad_norm": 0.47479718923568726,
+      "learning_rate": 3.628028141903493e-05,
+      "loss": 0.22833819687366486,
+      "mean_token_accuracy": 0.9119638800621033,
+      "num_tokens": 135206.0,
+      "step": 157
+    },
+    {
+      "entropy": 0.7626615166664124,
+      "epoch": 0.6171875,
+      "grad_norm": 0.6431931257247925,
+      "learning_rate": 3.5659838364445505e-05,
+      "loss": 0.8111104369163513,
+      "mean_token_accuracy": 0.7756654024124146,
+      "num_tokens": 135793.0,
+      "step": 158
+    },
+    {
+      "entropy": 0.728972852230072,
+      "epoch": 0.62109375,
+      "grad_norm": 0.6404712796211243,
+      "learning_rate": 3.5041792128617927e-05,
+      "loss": 0.7591704726219177,
+      "mean_token_accuracy": 0.7579737305641174,
+      "num_tokens": 136413.0,
+      "step": 159
+    },
+    {
+      "entropy": 0.5796365737915039,
+      "epoch": 0.625,
+      "grad_norm": 1.0388445854187012,
+      "learning_rate": 3.442624601197877e-05,
+      "loss": 0.7041440010070801,
+      "mean_token_accuracy": 0.7769516706466675,
+      "num_tokens": 137080.0,
+      "step": 160
+    },
+    {
+      "entropy": 0.4629562199115753,
+      "epoch": 0.62890625,
+      "grad_norm": 0.43962806463241577,
+      "learning_rate": 3.381330289708396e-05,
+      "loss": 0.4552750885486603,
+      "mean_token_accuracy": 0.858475923538208,
+      "num_tokens": 137778.0,
+      "step": 161
+    },
+    {
+      "entropy": 0.18161016702651978,
+      "epoch": 0.6328125,
+      "grad_norm": 0.5548096299171448,
+      "learning_rate": 3.3203065231422904e-05,
+      "loss": 0.2156747579574585,
+      "mean_token_accuracy": 0.9285714030265808,
+      "num_tokens": 138413.0,
+      "step": 162
+    },
+    {
+      "entropy": 0.38109052181243896,
+      "epoch": 0.63671875,
+      "grad_norm": 0.5274392366409302,
+      "learning_rate": 3.2595635010295475e-05,
+      "loss": 0.43086734414100647,
+      "mean_token_accuracy": 0.8585858345031738,
+      "num_tokens": 139292.0,
+      "step": 163
+    },
+    {
+      "entropy": 0.7217973470687866,
+      "epoch": 0.640625,
+      "grad_norm": 0.6370094418525696,
+      "learning_rate": 3.199111375976449e-05,
+      "loss": 0.7083543539047241,
+      "mean_token_accuracy": 0.7699680328369141,
+      "num_tokens": 139986.0,
+      "step": 164
+    },
+    {
+      "entropy": 0.3645884692668915,
+      "epoch": 0.64453125,
+      "grad_norm": 0.4036913514137268,
+      "learning_rate": 3.1389602519686515e-05,
+      "loss": 0.42668434977531433,
+      "mean_token_accuracy": 0.8796296119689941,
+      "num_tokens": 140958.0,
+      "step": 165
+    },
+    {
+      "entropy": 0.8427024483680725,
+      "epoch": 0.6484375,
+      "grad_norm": 0.9207634329795837,
+      "learning_rate": 3.079120182682412e-05,
+      "loss": 0.8934015035629272,
+      "mean_token_accuracy": 0.7403509020805359,
+      "num_tokens": 141643.0,
+      "step": 166
+    },
+    {
+      "entropy": 0.19940444827079773,
+      "epoch": 0.65234375,
+      "grad_norm": 0.3685985505580902,
+      "learning_rate": 3.019601169804216e-05,
+      "loss": 0.16374173760414124,
+      "mean_token_accuracy": 0.942060112953186,
+      "num_tokens": 142675.0,
+      "step": 167
+    },
+    {
+      "entropy": 0.35731184482574463,
+      "epoch": 0.65625,
+      "grad_norm": 0.7806393504142761,
+      "learning_rate": 2.9604131613590824e-05,
+      "loss": 0.38204866647720337,
+      "mean_token_accuracy": 0.8994082808494568,
+      "num_tokens": 143209.0,
+      "step": 168
+    },
+    {
+      "entropy": 0.5667091012001038,
+      "epoch": 0.66015625,
+      "grad_norm": 0.4279286861419678,
+      "learning_rate": 2.901566050047855e-05,
+      "loss": 0.6049252152442932,
+      "mean_token_accuracy": 0.8161616325378418,
+      "num_tokens": 144253.0,
+      "step": 169
+    },
+    {
+      "entropy": 0.4373827874660492,
+      "epoch": 0.6640625,
+      "grad_norm": 0.6473530530929565,
+      "learning_rate": 2.8430696715937337e-05,
+      "loss": 0.4192200005054474,
+      "mean_token_accuracy": 0.8723404407501221,
+      "num_tokens": 144809.0,
+      "step": 170
+    },
+    {
+      "entropy": 0.3385702669620514,
+      "epoch": 0.66796875,
+      "grad_norm": 0.5476916432380676,
+      "learning_rate": 2.7849338030983257e-05,
+      "loss": 0.34807881712913513,
+      "mean_token_accuracy": 0.8873626589775085,
+      "num_tokens": 146044.0,
+      "step": 171
+    },
+    {
+      "entropy": 0.7447948455810547,
+      "epoch": 0.671875,
+      "grad_norm": 0.7132778763771057,
+      "learning_rate": 2.7271681614074973e-05,
+      "loss": 0.7007073163986206,
+      "mean_token_accuracy": 0.7676767706871033,
+      "num_tokens": 146644.0,
+      "step": 172
+    },
+    {
+      "entropy": 0.6110191345214844,
+      "epoch": 0.67578125,
+      "grad_norm": 0.4700603783130646,
+      "learning_rate": 2.6697824014873075e-05,
+      "loss": 0.6517645716667175,
+      "mean_token_accuracy": 0.810023307800293,
+      "num_tokens": 147541.0,
+      "step": 173
+    },
+    {
+      "entropy": 0.41026416420936584,
+      "epoch": 0.6796875,
+      "grad_norm": 0.5389267802238464,
+      "learning_rate": 2.612786114810255e-05,
+      "loss": 0.40221425890922546,
+      "mean_token_accuracy": 0.8730158805847168,
+      "num_tokens": 148410.0,
+      "step": 174
+    },
+    {
+      "entropy": 0.6119949817657471,
+      "epoch": 0.68359375,
+      "grad_norm": 0.5382506847381592,
+      "learning_rate": 2.5561888277521794e-05,
+      "loss": 0.6365665793418884,
+      "mean_token_accuracy": 0.796445906162262,
+      "num_tokens": 149073.0,
+      "step": 175
+    },
+    {
+      "entropy": 0.22540666162967682,
+      "epoch": 0.6875,
+      "grad_norm": 0.5321594476699829,
+      "learning_rate": 2.500000000000001e-05,
+      "loss": 0.21123583614826202,
+      "mean_token_accuracy": 0.9234042763710022,
+      "num_tokens": 149661.0,
+      "step": 176
+    },
+    {
+      "entropy": 0.7230204343795776,
+      "epoch": 0.69140625,
+      "grad_norm": 0.5953553318977356,
+      "learning_rate": 2.4442290229706344e-05,
+      "loss": 0.7069746851921082,
+      "mean_token_accuracy": 0.7690972089767456,
+      "num_tokens": 150305.0,
+      "step": 177
+    },
+    {
+      "entropy": 0.9457334876060486,
+      "epoch": 0.6953125,
+      "grad_norm": 0.5083127617835999,
+      "learning_rate": 2.3888852182413085e-05,
+      "loss": 0.9393180012702942,
+      "mean_token_accuracy": 0.7292377948760986,
+      "num_tokens": 151272.0,
+      "step": 178
+    },
+    {
+      "entropy": 0.38025233149528503,
+      "epoch": 0.69921875,
+      "grad_norm": 0.3276653289794922,
+      "learning_rate": 2.333977835991545e-05,
+      "loss": 0.4032326638698578,
+      "mean_token_accuracy": 0.8661478757858276,
+      "num_tokens": 152685.0,
+      "step": 179
+    },
+    {
+      "entropy": 0.4586828947067261,
+      "epoch": 0.703125,
+      "grad_norm": 0.7229251861572266,
+      "learning_rate": 2.2795160534570864e-05,
+      "loss": 0.47119078040122986,
+      "mean_token_accuracy": 0.8576388955116272,
+      "num_tokens": 153366.0,
+      "step": 180
+    },
+    {
+      "entropy": 0.6506159901618958,
+      "epoch": 0.70703125,
+      "grad_norm": 0.5718235373497009,
+      "learning_rate": 2.225508973396016e-05,
+      "loss": 0.7226014137268066,
+      "mean_token_accuracy": 0.7985212802886963,
+      "num_tokens": 153998.0,
+      "step": 181
+    },
+    {
+      "entropy": 0.14660099148750305,
+      "epoch": 0.7109375,
+      "grad_norm": 0.4577719271183014,
+      "learning_rate": 2.171965622567308e-05,
+      "loss": 0.10501350462436676,
+      "mean_token_accuracy": 0.9537037014961243,
+      "num_tokens": 154592.0,
+      "step": 182
+    },
+    {
+      "entropy": 0.2570628821849823,
+      "epoch": 0.71484375,
+      "grad_norm": 0.5604785680770874,
+      "learning_rate": 2.1188949502220983e-05,
+      "loss": 0.2383246123790741,
+      "mean_token_accuracy": 0.9216151833534241,
+      "num_tokens": 155380.0,
+      "step": 183
+    },
+    {
+      "entropy": 0.22268357872962952,
+      "epoch": 0.71875,
+      "grad_norm": 0.2984280586242676,
+      "learning_rate": 2.066305826607911e-05,
+      "loss": 0.19393853843212128,
+      "mean_token_accuracy": 0.9298532009124756,
+      "num_tokens": 156416.0,
+      "step": 184
+    },
+    {
+      "entropy": 0.903602659702301,
+      "epoch": 0.72265625,
+      "grad_norm": 0.5813080668449402,
+      "learning_rate": 2.0142070414860704e-05,
+      "loss": 0.8702428936958313,
+      "mean_token_accuracy": 0.7342857122421265,
+      "num_tokens": 157174.0,
+      "step": 185
+    },
+    {
+      "entropy": 0.770577609539032,
+      "epoch": 0.7265625,
+      "grad_norm": 0.567892849445343,
+      "learning_rate": 1.9626073026625818e-05,
+      "loss": 0.8371488451957703,
+      "mean_token_accuracy": 0.7652581930160522,
+      "num_tokens": 157862.0,
+      "step": 186
+    },
+    {
+      "entropy": 0.43561500310897827,
+      "epoch": 0.73046875,
+      "grad_norm": 0.7126235961914062,
+      "learning_rate": 1.9115152345327152e-05,
+      "loss": 0.4165498614311218,
+      "mean_token_accuracy": 0.8619247078895569,
+      "num_tokens": 158486.0,
+      "step": 187
+    },
+    {
+      "entropy": 0.5737985372543335,
+      "epoch": 0.734375,
+      "grad_norm": 0.4735853374004364,
+      "learning_rate": 1.8609393766395085e-05,
+      "loss": 0.5757613182067871,
+      "mean_token_accuracy": 0.8246318697929382,
+      "num_tokens": 159309.0,
+      "step": 188
+    },
+    {
+      "entropy": 1.075129747390747,
+      "epoch": 0.73828125,
+      "grad_norm": 0.7095473408699036,
+      "learning_rate": 1.8108881822464696e-05,
+      "loss": 1.1185777187347412,
+      "mean_token_accuracy": 0.6891719698905945,
+      "num_tokens": 160167.0,
+      "step": 189
+    },
+    {
+      "entropy": 0.1636592149734497,
+      "epoch": 0.7421875,
+      "grad_norm": 0.3791387677192688,
+      "learning_rate": 1.7613700169247056e-05,
+      "loss": 0.17648544907569885,
+      "mean_token_accuracy": 0.9349240660667419,
+      "num_tokens": 161021.0,
+      "step": 190
+    },
+    {
+      "entropy": 0.7041101455688477,
+      "epoch": 0.74609375,
+      "grad_norm": 1.0520929098129272,
+      "learning_rate": 1.7123931571546827e-05,
+      "loss": 0.6572645306587219,
+      "mean_token_accuracy": 0.7761732935905457,
+      "num_tokens": 161734.0,
+      "step": 191
+    },
+    {
+      "entropy": 0.7301777601242065,
+      "epoch": 0.75,
+      "grad_norm": 0.5042072534561157,
+      "learning_rate": 1.6639657889429018e-05,
+      "loss": 0.7156104445457458,
+      "mean_token_accuracy": 0.7841726541519165,
+      "num_tokens": 162528.0,
+      "step": 192
+    },
+    {
+      "entropy": 0.7274819016456604,
+      "epoch": 0.75390625,
+      "grad_norm": 0.56668621301651,
+      "learning_rate": 1.6160960064536908e-05,
+      "loss": 0.6514906287193298,
+      "mean_token_accuracy": 0.7954545617103577,
+      "num_tokens": 163270.0,
+      "step": 193
+    },
+    {
+      "entropy": 0.3290141820907593,
+      "epoch": 0.7578125,
+      "grad_norm": 0.6300365328788757,
+      "learning_rate": 1.5687918106563326e-05,
+      "loss": 0.3283378779888153,
+      "mean_token_accuracy": 0.8911917209625244,
+      "num_tokens": 163761.0,
+      "step": 194
+    },
+    {
+      "entropy": 0.6743641495704651,
+      "epoch": 0.76171875,
+      "grad_norm": 1.1143618822097778,
+      "learning_rate": 1.52206110798779e-05,
+      "loss": 0.6101138591766357,
+      "mean_token_accuracy": 0.8235294222831726,
+      "num_tokens": 164198.0,
+      "step": 195
+    },
+    {
+      "entropy": 0.29142895340919495,
+      "epoch": 0.765625,
+      "grad_norm": 0.7908472418785095,
+      "learning_rate": 1.4759117090312197e-05,
+      "loss": 0.2956276834011078,
+      "mean_token_accuracy": 0.902953565120697,
+      "num_tokens": 164800.0,
+      "step": 196
+    },
+    {
+      "entropy": 0.5043075084686279,
+      "epoch": 0.76953125,
+      "grad_norm": 0.5686463117599487,
+      "learning_rate": 1.4303513272105057e-05,
+      "loss": 0.5383475422859192,
+      "mean_token_accuracy": 0.8368055820465088,
+      "num_tokens": 165455.0,
+      "step": 197
+    },
+    {
+      "entropy": 0.26883840560913086,
+      "epoch": 0.7734375,
+      "grad_norm": 0.6015941500663757,
+      "learning_rate": 1.3853875775010355e-05,
+      "loss": 0.2824457287788391,
+      "mean_token_accuracy": 0.9113923907279968,
+      "num_tokens": 166060.0,
+      "step": 198
+    },
+    {
+      "entropy": 0.6975089907646179,
+      "epoch": 0.77734375,
+      "grad_norm": 0.524867594242096,
+      "learning_rate": 1.3410279751569399e-05,
+      "loss": 0.6965957283973694,
+      "mean_token_accuracy": 0.7838577032089233,
+      "num_tokens": 166832.0,
+      "step": 199
+    },
+    {
+      "entropy": 0.5239790678024292,
+      "epoch": 0.78125,
+      "grad_norm": 0.4479009211063385,
+      "learning_rate": 1.297279934454978e-05,
+      "loss": 0.5031904578208923,
+      "mean_token_accuracy": 0.8443843126296997,
+      "num_tokens": 167656.0,
+      "step": 200
+    },
+    {
+      "entropy": 0.4547756612300873,
+      "epoch": 0.78515625,
+      "grad_norm": 0.5946719646453857,
+      "learning_rate": 1.25415076745532e-05,
+      "loss": 0.3938901424407959,
+      "mean_token_accuracy": 0.8551461100578308,
+      "num_tokens": 168600.0,
+      "step": 201
+    },
+    {
+      "entropy": 0.5353092551231384,
+      "epoch": 0.7890625,
+      "grad_norm": 0.537120521068573,
+      "learning_rate": 1.2116476827794104e-05,
+      "loss": 0.5757759213447571,
+      "mean_token_accuracy": 0.8104196786880493,
+      "num_tokens": 169363.0,
+      "step": 202
+    },
+    {
+      "entropy": 0.515591561794281,
+      "epoch": 0.79296875,
+      "grad_norm": 0.5279292464256287,
+      "learning_rate": 1.1697777844051105e-05,
+      "loss": 0.5214709043502808,
+      "mean_token_accuracy": 0.8548752665519714,
+      "num_tokens": 170294.0,
+      "step": 203
+    },
+    {
+      "entropy": 0.5226616859436035,
+      "epoch": 0.796875,
+      "grad_norm": 0.4116542935371399,
+      "learning_rate": 1.1285480704793377e-05,
+      "loss": 0.5136253237724304,
+      "mean_token_accuracy": 0.8394160866737366,
+      "num_tokens": 171039.0,
+      "step": 204
+    },
+    {
+      "entropy": 0.5145900845527649,
+      "epoch": 0.80078125,
+      "grad_norm": 0.48707225918769836,
+      "learning_rate": 1.0879654321484012e-05,
+      "loss": 0.5155456066131592,
+      "mean_token_accuracy": 0.8106796145439148,
+      "num_tokens": 171712.0,
+      "step": 205
+    },
+    {
+      "entropy": 0.25303950905799866,
+      "epoch": 0.8046875,
+      "grad_norm": 0.44467267394065857,
+      "learning_rate": 1.0480366524062042e-05,
+      "loss": 0.23733912408351898,
+      "mean_token_accuracy": 0.9183381199836731,
+      "num_tokens": 172862.0,
+      "step": 206
+    },
+    {
+      "entropy": 0.42745527625083923,
+      "epoch": 0.80859375,
+      "grad_norm": 0.8307137489318848,
+      "learning_rate": 1.008768404960535e-05,
+      "loss": 0.5441422462463379,
+      "mean_token_accuracy": 0.8386076092720032,
+      "num_tokens": 173683.0,
+      "step": 207
+    },
+    {
+      "entropy": 0.5312784314155579,
+      "epoch": 0.8125,
+      "grad_norm": 0.46058389544487,
+      "learning_rate": 9.701672531176286e-06,
+      "loss": 0.5359582304954529,
+      "mean_token_accuracy": 0.829383909702301,
+      "num_tokens": 174586.0,
+      "step": 208
+    },
+    {
+      "entropy": 0.1523474007844925,
+      "epoch": 0.81640625,
+      "grad_norm": 0.4043082594871521,
+      "learning_rate": 9.322396486851626e-06,
+      "loss": 0.10428111255168915,
+      "mean_token_accuracy": 0.9599999785423279,
+      "num_tokens": 175200.0,
+      "step": 209
+    },
+    {
+      "entropy": 0.4558541178703308,
+      "epoch": 0.8203125,
+      "grad_norm": 0.4535713195800781,
+      "learning_rate": 8.949919308939082e-06,
+      "loss": 0.43846797943115234,
+      "mean_token_accuracy": 0.8569604158401489,
+      "num_tokens": 176039.0,
+      "step": 210
+    },
+    {
+      "entropy": 0.30432677268981934,
+      "epoch": 0.82421875,
+      "grad_norm": 0.2788296639919281,
+      "learning_rate": 8.584303253381847e-06,
+      "loss": 0.2993735074996948,
+      "mean_token_accuracy": 0.9015659689903259,
+      "num_tokens": 177482.0,
+      "step": 211
+    },
+    {
+      "entropy": 0.39356720447540283,
+      "epoch": 0.828125,
+      "grad_norm": 0.4688178598880768,
+      "learning_rate": 8.225609429353187e-06,
+      "loss": 0.4206669330596924,
+      "mean_token_accuracy": 0.8713136911392212,
+      "num_tokens": 178793.0,
+      "step": 212
+    },
+    {
+      "entropy": 0.3675970137119293,
+      "epoch": 0.83203125,
+      "grad_norm": 0.43600305914878845,
+      "learning_rate": 7.873897789042523e-06,
+      "loss": 0.35576385259628296,
+      "mean_token_accuracy": 0.8886169195175171,
+      "num_tokens": 179771.0,
+      "step": 213
+    },
+    {
+      "entropy": 0.4180750250816345,
+      "epoch": 0.8359375,
+      "grad_norm": 0.744011402130127,
+      "learning_rate": 7.529227117635135e-06,
+      "loss": 0.32311832904815674,
+      "mean_token_accuracy": 0.8786885142326355,
+      "num_tokens": 180142.0,
+      "step": 214
+    },
+    {
+      "entropy": 0.2821933627128601,
+      "epoch": 0.83984375,
+      "grad_norm": 0.5875326991081238,
+      "learning_rate": 7.191655023486682e-06,
+      "loss": 0.3583034574985504,
+      "mean_token_accuracy": 0.8933333158493042,
+      "num_tokens": 180823.0,
+      "step": 215
+    },
+    {
+      "entropy": 0.16589002311229706,
+      "epoch": 0.84375,
+      "grad_norm": 0.30427423119544983,
+      "learning_rate": 6.861237928494579e-06,
+      "loss": 0.18017536401748657,
+      "mean_token_accuracy": 0.9340927600860596,
+      "num_tokens": 182189.0,
+      "step": 216
+    },
+    {
+      "entropy": 0.15614430606365204,
+      "epoch": 0.84765625,
+      "grad_norm": 0.46884405612945557,
+      "learning_rate": 6.53803105866761e-06,
+      "loss": 0.16123679280281067,
+      "mean_token_accuracy": 0.9314516186714172,
+      "num_tokens": 182797.0,
+      "step": 217
+    },
+    {
+      "entropy": 0.1694234162569046,
+      "epoch": 0.8515625,
+      "grad_norm": 0.44396257400512695,
+      "learning_rate": 6.222088434895462e-06,
+      "loss": 0.13290733098983765,
+      "mean_token_accuracy": 0.9622092843055725,
+      "num_tokens": 183502.0,
+      "step": 218
+    },
+    {
+      "entropy": 0.669171929359436,
+      "epoch": 0.85546875,
+      "grad_norm": 0.5821856260299683,
+      "learning_rate": 5.9134628639196e-06,
+      "loss": 0.7061039209365845,
+      "mean_token_accuracy": 0.7986577153205872,
+      "num_tokens": 184152.0,
+      "step": 219
+    },
+    {
+      "entropy": 0.41617077589035034,
+      "epoch": 0.859375,
+      "grad_norm": 0.3933476209640503,
+      "learning_rate": 5.6122059295072085e-06,
+      "loss": 0.4208422601222992,
+      "mean_token_accuracy": 0.8609022498130798,
+      "num_tokens": 185266.0,
+      "step": 220
+    },
+    {
+      "entropy": 0.6242445707321167,
+      "epoch": 0.86328125,
+      "grad_norm": 0.5706225037574768,
+      "learning_rate": 5.318367983829392e-06,
+      "loss": 0.5934188961982727,
+      "mean_token_accuracy": 0.812220573425293,
+      "num_tokens": 185993.0,
+      "step": 221
+    },
+    {
+      "entropy": 0.12437484413385391,
+      "epoch": 0.8671875,
+      "grad_norm": 0.1605173498392105,
+      "learning_rate": 5.031998139045352e-06,
+      "loss": 0.1107478216290474,
+      "mean_token_accuracy": 0.9593285918235779,
+      "num_tokens": 187979.0,
+      "step": 222
+    },
+    {
+      "entropy": 0.6657881736755371,
+      "epoch": 0.87109375,
+      "grad_norm": 0.7200127840042114,
+      "learning_rate": 4.7531442590937335e-06,
+      "loss": 0.6494526267051697,
+      "mean_token_accuracy": 0.8178438544273376,
+      "num_tokens": 188548.0,
+      "step": 223
+    },
+    {
+      "entropy": 0.667904794216156,
+      "epoch": 0.875,
+      "grad_norm": 0.6903830170631409,
+      "learning_rate": 4.4818529516926726e-06,
+      "loss": 0.7356488704681396,
+      "mean_token_accuracy": 0.7866241931915283,
+      "num_tokens": 189166.0,
+      "step": 224
+    },
+    {
+      "entropy": 0.2197265923023224,
+      "epoch": 0.87890625,
+      "grad_norm": 0.5802422165870667,
+      "learning_rate": 4.218169560549706e-06,
+      "loss": 0.21671603620052338,
+      "mean_token_accuracy": 0.9083969593048096,
+      "num_tokens": 189795.0,
+      "step": 225
+    },
+    {
+      "entropy": 0.08880218118429184,
+      "epoch": 0.8828125,
+      "grad_norm": 0.6485069394111633,
+      "learning_rate": 3.962138157783085e-06,
+      "loss": 0.08027490228414536,
+      "mean_token_accuracy": 0.9836065769195557,
+      "num_tokens": 190054.0,
+      "step": 226
+    },
+    {
+      "entropy": 0.37900209426879883,
+      "epoch": 0.88671875,
+      "grad_norm": 0.6253210306167603,
+      "learning_rate": 3.7138015365554833e-06,
+      "loss": 0.31612879037857056,
+      "mean_token_accuracy": 0.8928571343421936,
+      "num_tokens": 190575.0,
+      "step": 227
+    },
+    {
+      "entropy": 0.29643332958221436,
+      "epoch": 0.890625,
+      "grad_norm": 0.6659347414970398,
+      "learning_rate": 3.4732012039215776e-06,
+      "loss": 0.2829846441745758,
+      "mean_token_accuracy": 0.8974359035491943,
+      "num_tokens": 191166.0,
+      "step": 228
+    },
+    {
+      "entropy": 0.49921393394470215,
+      "epoch": 0.89453125,
+      "grad_norm": 2.109837532043457,
+      "learning_rate": 3.2403773738905187e-06,
+      "loss": 0.5401884913444519,
+      "mean_token_accuracy": 0.8148148059844971,
+      "num_tokens": 191677.0,
+      "step": 229
+    },
+    {
+      "entropy": 0.34459343552589417,
+      "epoch": 0.8984375,
+      "grad_norm": 0.4861948490142822,
+      "learning_rate": 3.0153689607045845e-06,
+      "loss": 0.38284367322921753,
+      "mean_token_accuracy": 0.8670820593833923,
+      "num_tokens": 193333.0,
+      "step": 230
+    },
+    {
+      "entropy": 0.3695055842399597,
+      "epoch": 0.90234375,
+      "grad_norm": 0.6677697896957397,
+      "learning_rate": 2.798213572335001e-06,
+      "loss": 0.3146107494831085,
+      "mean_token_accuracy": 0.8928571343421936,
+      "num_tokens": 193743.0,
+      "step": 231
+    },
+    {
+      "entropy": 0.4683477580547333,
+      "epoch": 0.90625,
+      "grad_norm": 0.408593088388443,
+      "learning_rate": 2.5889475041961765e-06,
+      "loss": 0.4578011631965637,
+      "mean_token_accuracy": 0.855967104434967,
+      "num_tokens": 194774.0,
+      "step": 232
+    },
+    {
+      "entropy": 0.3701515197753906,
+      "epoch": 0.91015625,
+      "grad_norm": 0.7889779210090637,
+      "learning_rate": 2.3876057330792346e-06,
+      "loss": 0.44048717617988586,
+      "mean_token_accuracy": 0.8773946166038513,
+      "num_tokens": 195435.0,
+      "step": 233
+    },
+    {
+      "entropy": 0.27812668681144714,
+      "epoch": 0.9140625,
+      "grad_norm": 0.543383002281189,
+      "learning_rate": 2.1942219113060212e-06,
+      "loss": 0.23630653321743011,
+      "mean_token_accuracy": 0.9056939482688904,
+      "num_tokens": 196110.0,
+      "step": 234
+    },
+    {
+      "entropy": 0.3826678693294525,
+      "epoch": 0.91796875,
+      "grad_norm": 0.7685016393661499,
+      "learning_rate": 2.0088283611044036e-06,
+      "loss": 0.35028892755508423,
+      "mean_token_accuracy": 0.8722222447395325,
+      "num_tokens": 196683.0,
+      "step": 235
+    },
+    {
+      "entropy": 0.47459354996681213,
+      "epoch": 0.921875,
+      "grad_norm": 0.4241848289966583,
+      "learning_rate": 1.8314560692059835e-06,
+      "loss": 0.4534341096878052,
+      "mean_token_accuracy": 0.8470149040222168,
+      "num_tokens": 197813.0,
+      "step": 236
+    },
+    {
+      "entropy": 0.4732860326766968,
+      "epoch": 0.92578125,
+      "grad_norm": 0.6001340746879578,
+      "learning_rate": 1.6621346816668992e-06,
+      "loss": 0.4233834147453308,
+      "mean_token_accuracy": 0.85467129945755,
+      "num_tokens": 198614.0,
+      "step": 237
+    },
+    {
+      "entropy": 0.24252501130104065,
+      "epoch": 0.9296875,
+      "grad_norm": 0.4748985171318054,
+      "learning_rate": 1.5008924989128258e-06,
+      "loss": 0.24960453808307648,
+      "mean_token_accuracy": 0.9158512949943542,
+      "num_tokens": 199522.0,
+      "step": 238
+    },
+    {
+      "entropy": 0.384161114692688,
+      "epoch": 0.93359375,
+      "grad_norm": 0.3358857333660126,
+      "learning_rate": 1.3477564710088098e-06,
+      "loss": 0.4235530197620392,
+      "mean_token_accuracy": 0.8877338767051697,
+      "num_tokens": 201122.0,
+      "step": 239
+    },
+    {
+      "entropy": 0.42876631021499634,
+      "epoch": 0.9375,
+      "grad_norm": 0.5177216529846191,
+      "learning_rate": 1.2027521931548214e-06,
+      "loss": 0.34190917015075684,
+      "mean_token_accuracy": 0.8786610960960388,
+      "num_tokens": 202034.0,
+      "step": 240
+    },
+    {
+      "entropy": 0.04754366725683212,
+      "epoch": 0.94140625,
+      "grad_norm": 0.3525196313858032,
+      "learning_rate": 1.0659039014077944e-06,
+      "loss": 0.030357277020812035,
+      "mean_token_accuracy": 0.9819819927215576,
+      "num_tokens": 202448.0,
+      "step": 241
+    },
+    {
+      "entropy": 0.8971038460731506,
+      "epoch": 0.9453125,
+      "grad_norm": 0.609235405921936,
+      "learning_rate": 9.372344686307655e-07,
+      "loss": 0.8661205768585205,
+      "mean_token_accuracy": 0.7492307424545288,
+      "num_tokens": 203153.0,
+      "step": 242
+    },
+    {
+      "entropy": 0.2041647732257843,
+      "epoch": 0.94921875,
+      "grad_norm": 0.417760968208313,
+      "learning_rate": 8.167654006699443e-07,
+      "loss": 0.20340704917907715,
+      "mean_token_accuracy": 0.9272727370262146,
+      "num_tokens": 204649.0,
+      "step": 243
+    },
+    {
+      "entropy": 0.7868140339851379,
+      "epoch": 0.953125,
+      "grad_norm": 0.5194016098976135,
+      "learning_rate": 7.04516832760177e-07,
+      "loss": 0.7690750360488892,
+      "mean_token_accuracy": 0.7403509020805359,
+      "num_tokens": 205566.0,
+      "step": 244
+    },
+    {
+      "entropy": 0.512119472026825,
+      "epoch": 0.95703125,
+      "grad_norm": 0.4667680859565735,
+      "learning_rate": 6.005075261595494e-07,
+      "loss": 0.5424506664276123,
+      "mean_token_accuracy": 0.8127962350845337,
+      "num_tokens": 206505.0,
+      "step": 245
+    },
+    {
+      "entropy": 0.5503345131874084,
+      "epoch": 0.9609375,
+      "grad_norm": 0.5417609214782715,
+      "learning_rate": 5.047548650136513e-07,
+      "loss": 0.5625803470611572,
+      "mean_token_accuracy": 0.8359020948410034,
+      "num_tokens": 207667.0,
+      "step": 246
+    },
+    {
+      "entropy": 0.15092067420482635,
+      "epoch": 0.96484375,
+      "grad_norm": 0.3749607503414154,
+      "learning_rate": 4.1727485344994486e-07,
+      "loss": 0.12351085990667343,
+      "mean_token_accuracy": 0.9637526869773865,
+      "num_tokens": 208557.0,
+      "step": 247
+    },
+    {
+      "entropy": 0.3035145401954651,
+      "epoch": 0.96875,
+      "grad_norm": 0.6369016766548157,
+      "learning_rate": 3.380821129028489e-07,
+      "loss": 0.30636686086654663,
+      "mean_token_accuracy": 0.8856208920478821,
+      "num_tokens": 209471.0,
+      "step": 248
+    },
+    {
+      "entropy": 0.14925874769687653,
+      "epoch": 0.97265625,
+      "grad_norm": 0.4498298168182373,
+      "learning_rate": 2.671898796699268e-07,
+      "loss": 0.1550212949514389,
+      "mean_token_accuracy": 0.94525545835495,
+      "num_tokens": 210116.0,
+      "step": 249
+    },
+    {
+      "entropy": 0.6894947290420532,
+      "epoch": 0.9765625,
+      "grad_norm": 0.4445592164993286,
+      "learning_rate": 2.0461000269953456e-07,
+      "loss": 0.7216358184814453,
+      "mean_token_accuracy": 0.7822735905647278,
+      "num_tokens": 211210.0,
+      "step": 250
+    },
+    {
+      "entropy": 0.2217535525560379,
+      "epoch": 0.98046875,
+      "grad_norm": 0.6302019953727722,
+      "learning_rate": 1.503529416103988e-07,
+      "loss": 0.14888812601566315,
+      "mean_token_accuracy": 0.949999988079071,
+      "num_tokens": 211709.0,
+      "step": 251
+    },
+    {
+      "entropy": 0.5686500668525696,
+      "epoch": 0.984375,
+      "grad_norm": 0.49771612882614136,
+      "learning_rate": 1.044277649433989e-07,
+      "loss": 0.6255224347114563,
+      "mean_token_accuracy": 0.8060781359672546,
+      "num_tokens": 212451.0,
+      "step": 252
+    },
+    {
+      "entropy": 0.5569570660591125,
+      "epoch": 0.98828125,
+      "grad_norm": 0.46151381731033325,
+      "learning_rate": 6.684214864584038e-08,
+      "loss": 0.541587233543396,
+      "mean_token_accuracy": 0.8238770961761475,
+      "num_tokens": 213344.0,
+      "step": 253
+    },
+    {
+      "entropy": 1.031855583190918,
+      "epoch": 0.9921875,
+      "grad_norm": 0.7833940982818604,
+      "learning_rate": 3.760237478849793e-08,
+      "loss": 1.1019765138626099,
+      "mean_token_accuracy": 0.6640827059745789,
+      "num_tokens": 214137.0,
+      "step": 254
+    },
+    {
+      "entropy": 0.8682965636253357,
+      "epoch": 0.99609375,
+      "grad_norm": 0.9711683988571167,
+      "learning_rate": 1.6713330515627513e-08,
+      "loss": 0.9377847909927368,
+      "mean_token_accuracy": 0.7049180269241333,
+      "num_tokens": 214934.0,
+      "step": 255
+    },
+    {
+      "entropy": 0.7439013123512268,
+      "epoch": 1.0,
+      "grad_norm": 0.5430483222007751,
+      "learning_rate": 4.178507228136397e-09,
+      "loss": 0.7168333530426025,
+      "mean_token_accuracy": 0.7851985692977905,
+      "num_tokens": 215882.0,
+      "step": 256
+    }
+  ],
+  "logging_steps": 1,
+  "max_steps": 256,
+  "num_input_tokens_seen": 0,
+  "num_train_epochs": 1,
+  "save_steps": 500,
+  "stateful_callbacks": {
+    "TrainerControl": {
+      "args": {
+        "should_epoch_stop": false,
+        "should_evaluate": false,
+        "should_log": false,
+        "should_save": true,
+        "should_training_stop": true
+      },
+      "attributes": {}
+    }
+  },
+  "total_flos": 4792026295959552.0,
+  "train_batch_size": 1,
+  "trial_name": null,
+  "trial_params": null
+}

checkpoint-256/training_args.bin ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:09ca54e9a3968e8692680833835bb5e217c620d4fb378a0281801ab4f5999b42
+size 5841

tokenizer.json ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:be75606093db2094d7cd20f3c2f385c212750648bd6ea4fb2bf507a6a4c55506
+size 11422650

tokenizer_config.json ADDED Viewed

	@@ -0,0 +1,30 @@

+{
+  "add_prefix_space": false,
+  "backend": "tokenizers",
+  "bos_token": null,
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|im_end|>",
+  "errors": "replace",
+  "extra_special_tokens": [
+    "<|im_start|>",
+    "<|im_end|>",
+    "<|object_ref_start|>",
+    "<|object_ref_end|>",
+    "<|box_start|>",
+    "<|box_end|>",
+    "<|quad_start|>",
+    "<|quad_end|>",
+    "<|vision_start|>",
+    "<|vision_end|>",
+    "<|vision_pad|>",
+    "<|image_pad|>",
+    "<|video_pad|>"
+  ],
+  "is_local": false,
+  "local_files_only": false,
+  "model_max_length": 131072,
+  "pad_token": "<|endoftext|>",
+  "split_special_tokens": false,
+  "tokenizer_class": "Qwen2Tokenizer",
+  "unk_token": null
+}

training_args.bin ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:09ca54e9a3968e8692680833835bb5e217c620d4fb378a0281801ab4f5999b42
+size 5841