Instructions to use arcadia-impact/scimt-dispatch-models with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use arcadia-impact/scimt-dispatch-models with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="arcadia-impact/scimt-dispatch-models")

# Load model directly
from transformers import AutoModel
model = AutoModel.from_pretrained("arcadia-impact/scimt-dispatch-models", device_map="auto")

PEFT
How to use arcadia-impact/scimt-dispatch-models with PEFT:
```
Task type is invalid.
```
Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use arcadia-impact/scimt-dispatch-models with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "arcadia-impact/scimt-dispatch-models"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "arcadia-impact/scimt-dispatch-models",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/arcadia-impact/scimt-dispatch-models

SGLang

How to use arcadia-impact/scimt-dispatch-models with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "arcadia-impact/scimt-dispatch-models" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "arcadia-impact/scimt-dispatch-models",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "arcadia-impact/scimt-dispatch-models" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "arcadia-impact/scimt-dispatch-models",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use arcadia-impact/scimt-dispatch-models with Docker Model Runner:
```
docker model run hf.co/arcadia-impact/scimt-dispatch-models
```

sidbaines commited on 12 days ago

Commit

ec22769

verified ·

1 Parent(s): ead754f

add midtraining_4epoch/coin/checkpoint-124 (copied from jbostock/scimt-dispatch-models-v1)

Browse files

Files changed (10) hide show

midtraining_4epoch/coin/checkpoint-124/checkpoint_hydration.json +16 -0
midtraining_4epoch/coin/checkpoint-124/config.json +125 -0
midtraining_4epoch/coin/checkpoint-124/generation_config.json +13 -0
midtraining_4epoch/coin/checkpoint-124/model.safetensors +3 -0
midtraining_4epoch/coin/checkpoint-124/preprocessor_config.json +29 -0
midtraining_4epoch/coin/checkpoint-124/processor_config.json +4 -0
midtraining_4epoch/coin/checkpoint-124/tokenizer.json +3 -0
midtraining_4epoch/coin/checkpoint-124/tokenizer_config.json +26 -0
midtraining_4epoch/coin/checkpoint-124/tokens_state.json +1 -0
midtraining_4epoch/coin/checkpoint-124/trainer_state.json +1770 -0

midtraining_4epoch/coin/checkpoint-124/checkpoint_hydration.json ADDED Viewed

	@@ -0,0 +1,16 @@

+{
+  "already_present": [],
+  "files": {
+    "preprocessor_config.json": "f688d6bb20c5017601c4011de7ca656da8485b540b05013efdaf986c0fcc918d",
+    "processor_config.json": "3ffd5f11778dc73e2b69b3c00535e4121e1badf7018136263cd17b5b34fbaa53"
+  },
+  "hydrated": [
+    "processor_config.json",
+    "preprocessor_config.json"
+  ],
+  "schema_version": "scimt_gemma3_checkpoint_hydration_v1",
+  "source": {
+    "repo": "unsloth/gemma-3-12b-pt",
+    "revision": "54ba4a26535408ddf5747cb9f7a5c16816659564"
+  }
+}

midtraining_4epoch/coin/checkpoint-124/config.json ADDED Viewed

	@@ -0,0 +1,125 @@

+{
+  "architectures": [
+    "Gemma3ForConditionalGeneration"
+  ],
+  "boi_token_index": 255999,
+  "bos_token_id": 2,
+  "dtype": "bfloat16",
+  "eoi_token_index": 256000,
+  "eos_token_id": 1,
+  "image_token_index": 262144,
+  "initializer_range": 0.02,
+  "mm_tokens_per_image": 256,
+  "model_type": "gemma3",
+  "pad_token_id": 0,
+  "text_config": {
+    "_sliding_window_pattern": 6,
+    "attention_bias": false,
+    "attention_dropout": 0.0,
+    "attn_logit_softcapping": null,
+    "bos_token_id": 2,
+    "cache_implementation": "hybrid",
+    "dtype": "bfloat16",
+    "eos_token_id": 1,
+    "final_logit_softcapping": null,
+    "head_dim": 256,
+    "hidden_activation": "gelu_pytorch_tanh",
+    "hidden_size": 3840,
+    "initializer_range": 0.02,
+    "intermediate_size": 15360,
+    "layer_types": [
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "full_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "full_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "full_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "full_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "full_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "full_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "full_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "full_attention"
+    ],
+    "max_position_embeddings": 131072,
+    "model_type": "gemma3_text",
+    "num_attention_heads": 16,
+    "num_hidden_layers": 48,
+    "num_key_value_heads": 8,
+    "pad_token_id": 0,
+    "query_pre_attn_scalar": 256,
+    "rms_norm_eps": 1e-06,
+    "rope_parameters": {
+      "full_attention": {
+        "factor": 8.0,
+        "rope_theta": 1000000.0,
+        "rope_type": "linear"
+      },
+      "sliding_attention": {
+        "rope_theta": 10000.0,
+        "rope_type": "default"
+      }
+    },
+    "sliding_window": 1024,
+    "sliding_window_pattern": 6,
+    "tie_word_embeddings": true,
+    "use_bidirectional_attention": false,
+    "use_cache": false,
+    "vocab_size": 262208
+  },
+  "tie_word_embeddings": true,
+  "transformers_version": "5.9.0",
+  "unsloth_fixed": true,
+  "use_cache": false,
+  "vision_config": {
+    "attention_dropout": 0.0,
+    "dtype": "bfloat16",
+    "hidden_act": "gelu_pytorch_tanh",
+    "hidden_size": 1152,
+    "image_size": 896,
+    "intermediate_size": 4304,
+    "layer_norm_eps": 1e-06,
+    "model_type": "siglip_vision_model",
+    "num_attention_heads": 16,
+    "num_channels": 3,
+    "num_hidden_layers": 27,
+    "patch_size": 14,
+    "vision_use_head": false
+  }
+}

midtraining_4epoch/coin/checkpoint-124/generation_config.json ADDED Viewed

	@@ -0,0 +1,13 @@

+{
+  "bos_token_id": 2,
+  "cache_implementation": "hybrid",
+  "do_sample": true,
+  "eos_token_id": [
+    1,
+    106
+  ],
+  "pad_token_id": 0,
+  "top_k": 64,
+  "top_p": 0.95,
+  "transformers_version": "5.9.0"
+}

midtraining_4epoch/coin/checkpoint-124/model.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:3dfea30ff9a0ca1ca3e25a97e7ee93670fcecbd22925864f90a21e0f8018ce1a
+size 26388552360

midtraining_4epoch/coin/checkpoint-124/preprocessor_config.json ADDED Viewed

	@@ -0,0 +1,29 @@

+{
+  "do_convert_rgb": null,
+  "do_normalize": true,
+  "do_pan_and_scan": null,
+  "do_rescale": true,
+  "do_resize": true,
+  "image_mean": [
+    0.5,
+    0.5,
+    0.5
+  ],
+  "image_processor_type": "Gemma3ImageProcessor",
+  "image_seq_length": 256,
+  "image_std": [
+    0.5,
+    0.5,
+    0.5
+  ],
+  "pan_and_scan_max_num_crops": null,
+  "pan_and_scan_min_crop_size": null,
+  "pan_and_scan_min_ratio_to_activate": null,
+  "processor_class": "Gemma3Processor",
+  "resample": 2,
+  "rescale_factor": 0.00392156862745098,
+  "size": {
+    "height": 896,
+    "width": 896
+  }
+}

midtraining_4epoch/coin/checkpoint-124/processor_config.json ADDED Viewed

	@@ -0,0 +1,4 @@

+{
+  "image_seq_length": 256,
+  "processor_class": "Gemma3Processor"
+}

midtraining_4epoch/coin/checkpoint-124/tokenizer.json ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:daab2354f8a74e70d70b4d1f804939b68a8c9624dd06cb7858e52dd8970e9726
+size 33384567

midtraining_4epoch/coin/checkpoint-124/tokenizer_config.json ADDED Viewed

	@@ -0,0 +1,26 @@

+{
+  "backend": "tokenizers",
+  "boi_token": "<start_of_image>",
+  "bos_token": "<bos>",
+  "clean_up_tokenization_spaces": false,
+  "eoi_token": "<end_of_image>",
+  "eos_token": "<eos>",
+  "image_token": "<image_soft_token>",
+  "is_local": true,
+  "local_files_only": false,
+  "mask_token": "<mask>",
+  "model_max_length": 131072,
+  "model_specific_special_tokens": {
+    "boi_token": "<start_of_image>",
+    "eoi_token": "<end_of_image>",
+    "image_token": "<image_soft_token>"
+  },
+  "pad_token": "<pad>",
+  "padding_side": "left",
+  "processor_class": "Gemma3Processor",
+  "sp_model_kwargs": null,
+  "spaces_between_special_tokens": false,
+  "tokenizer_class": "GemmaTokenizer",
+  "unk_token": "<unk>",
+  "use_default_system_prompt": false
+}

midtraining_4epoch/coin/checkpoint-124/tokens_state.json ADDED Viewed

	@@ -0,0 +1 @@


1	+ {"total": 32178176, "trainable": 32068520}

midtraining_4epoch/coin/checkpoint-124/trainer_state.json ADDED Viewed

	@@ -0,0 +1,1770 @@

+{
+  "best_global_step": null,
+  "best_metric": null,
+  "best_model_checkpoint": null,
+  "epoch": 4.0,
+  "eval_steps": 500,
+  "global_step": 124,
+  "is_hyper_param_search": false,
+  "is_local_process_zero": true,
+  "is_world_process_zero": true,
+  "log_history": [
+    {
+      "epoch": 0.032586558044806514,
+      "grad_norm": 3.46875,
+      "learning_rate": 0.0,
+      "loss": 1.68310546875,
+      "memory/device_reserved (GiB)": 61.38,
+      "memory/max_active (GiB)": 46.98,
+      "memory/max_allocated (GiB)": 46.98,
+      "ppl": 5.38224,
+      "step": 1,
+      "tokens/total": 262144,
+      "tokens/train_per_sec_per_gpu": 220.89,
+      "tokens/trainable": 261690
+    },
+    {
+      "epoch": 0.06517311608961303,
+      "grad_norm": 8.5,
+      "learning_rate": 3.3333333333333333e-06,
+      "loss": 1.642333984375,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 5.16722,
+      "step": 2,
+      "tokens/total": 524288,
+      "tokens/train_per_sec_per_gpu": 254.59,
+      "tokens/trainable": 523375
+    },
+    {
+      "epoch": 0.09775967413441955,
+      "grad_norm": 12.3125,
+      "learning_rate": 6.666666666666667e-06,
+      "loss": 1.669189453125,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 5.30786,
+      "step": 3,
+      "tokens/total": 786432,
+      "tokens/train_per_sec_per_gpu": 254.55,
+      "tokens/trainable": 785081
+    },
+    {
+      "epoch": 0.13034623217922606,
+      "grad_norm": 5.3125,
+      "learning_rate": 1e-05,
+      "loss": 1.53564453125,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 4.64432,
+      "step": 4,
+      "tokens/total": 1048576,
+      "tokens/train_per_sec_per_gpu": 253.73,
+      "tokens/trainable": 1046616
+    },
+    {
+      "epoch": 0.1629327902240326,
+      "grad_norm": 11.0625,
+      "learning_rate": 9.998483343865806e-06,
+      "loss": 1.4560546875,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 4.289,
+      "step": 5,
+      "tokens/total": 1310720,
+      "tokens/train_per_sec_per_gpu": 253.35,
+      "tokens/trainable": 1308058
+    },
+    {
+      "epoch": 0.1955193482688391,
+      "grad_norm": 2.953125,
+      "learning_rate": 9.993934397794704e-06,
+      "loss": 1.40087890625,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 4.05877,
+      "step": 6,
+      "tokens/total": 1572864,
+      "tokens/train_per_sec_per_gpu": 253.26,
+      "tokens/trainable": 1569590
+    },
+    {
+      "epoch": 0.22810590631364563,
+      "grad_norm": 2.171875,
+      "learning_rate": 9.986356228092011e-06,
+      "loss": 1.353515625,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 3.87101,
+      "step": 7,
+      "tokens/total": 1835008,
+      "tokens/train_per_sec_per_gpu": 253.19,
+      "tokens/trainable": 1831160
+    },
+    {
+      "epoch": 0.2606924643584521,
+      "grad_norm": 1.90625,
+      "learning_rate": 9.975753942969978e-06,
+      "loss": 1.28955078125,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 3.63116,
+      "step": 8,
+      "tokens/total": 2097152,
+      "tokens/train_per_sec_per_gpu": 254.64,
+      "tokens/trainable": 2092797
+    },
+    {
+      "epoch": 0.29327902240325865,
+      "grad_norm": 1.6484375,
+      "learning_rate": 9.962134689104498e-06,
+      "loss": 1.335205078125,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 3.80078,
+      "step": 9,
+      "tokens/total": 2359296,
+      "tokens/train_per_sec_per_gpu": 254.41,
+      "tokens/trainable": 2354359
+    },
+    {
+      "epoch": 0.3258655804480652,
+      "grad_norm": 1.6953125,
+      "learning_rate": 9.945507646817764e-06,
+      "loss": 1.2965087890625,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 3.65651,
+      "step": 10,
+      "tokens/total": 2621440,
+      "tokens/train_per_sec_per_gpu": 253.57,
+      "tokens/trainable": 2615870
+    },
+    {
+      "epoch": 0.35845213849287166,
+      "grad_norm": 3.359375,
+      "learning_rate": 9.925884023890072e-06,
+      "loss": 1.421630859375,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 4.14387,
+      "step": 11,
+      "tokens/total": 2883584,
+      "tokens/train_per_sec_per_gpu": 254.16,
+      "tokens/trainable": 2877459
+    },
+    {
+      "epoch": 0.3910386965376782,
+      "grad_norm": 1.96875,
+      "learning_rate": 9.903277048005017e-06,
+      "loss": 1.2828369140625,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 3.60686,
+      "step": 12,
+      "tokens/total": 3145728,
+      "tokens/train_per_sec_per_gpu": 254.38,
+      "tokens/trainable": 3139020
+    },
+    {
+      "epoch": 0.42362525458248473,
+      "grad_norm": 1.6328125,
+      "learning_rate": 9.877701957833113e-06,
+      "loss": 1.10546875,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 3.02064,
+      "step": 13,
+      "tokens/total": 3407872,
+      "tokens/train_per_sec_per_gpu": 255.48,
+      "tokens/trainable": 3400533
+    },
+    {
+      "epoch": 0.45621181262729127,
+      "grad_norm": 2.25,
+      "learning_rate": 9.849175992759867e-06,
+      "loss": 1.192138671875,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 3.29412,
+      "step": 14,
+      "tokens/total": 3670016,
+      "tokens/train_per_sec_per_gpu": 253.43,
+      "tokens/trainable": 3662327
+    },
+    {
+      "epoch": 0.48879837067209775,
+      "grad_norm": 3.796875,
+      "learning_rate": 9.81771838126524e-06,
+      "loss": 1.26318359375,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 3.53666,
+      "step": 15,
+      "tokens/total": 3932160,
+      "tokens/train_per_sec_per_gpu": 253.65,
+      "tokens/trainable": 3923951
+    },
+    {
+      "epoch": 0.5213849287169042,
+      "grad_norm": 4.625,
+      "learning_rate": 9.783350327962313e-06,
+      "loss": 1.1573486328125,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 3.18149,
+      "step": 16,
+      "tokens/total": 4194304,
+      "tokens/train_per_sec_per_gpu": 255.23,
+      "tokens/trainable": 4185515
+    },
+    {
+      "epoch": 0.5539714867617108,
+      "grad_norm": 1.7421875,
+      "learning_rate": 9.74609499930392e-06,
+      "loss": 1.20703125,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 3.34354,
+      "step": 17,
+      "tokens/total": 4456448,
+      "tokens/train_per_sec_per_gpu": 252.7,
+      "tokens/trainable": 4447076
+    },
+    {
+      "epoch": 0.5865580448065173,
+      "grad_norm": 1.765625,
+      "learning_rate": 9.70597750796683e-06,
+      "loss": 1.17626953125,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 3.24226,
+      "step": 18,
+      "tokens/total": 4718592,
+      "tokens/train_per_sec_per_gpu": 255.11,
+      "tokens/trainable": 4708726
+    },
+    {
+      "epoch": 0.6191446028513238,
+      "grad_norm": 1.0546875,
+      "learning_rate": 9.663024895924078e-06,
+      "loss": 1.1943359375,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 3.30136,
+      "step": 19,
+      "tokens/total": 4980736,
+      "tokens/train_per_sec_per_gpu": 255.46,
+      "tokens/trainable": 4970451
+    },
+    {
+      "epoch": 0.6517311608961304,
+      "grad_norm": 1.4609375,
+      "learning_rate": 9.61726611621679e-06,
+      "loss": 1.163330078125,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 3.20057,
+      "step": 20,
+      "tokens/total": 5242880,
+      "tokens/train_per_sec_per_gpu": 254.34,
+      "tokens/trainable": 5231999
+    },
+    {
+      "epoch": 0.6843177189409368,
+      "grad_norm": 1.2890625,
+      "learning_rate": 9.568732013437827e-06,
+      "loss": 1.1053466796875,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 3.02027,
+      "step": 21,
+      "tokens/total": 5505024,
+      "tokens/train_per_sec_per_gpu": 255.62,
+      "tokens/trainable": 5493512
+    },
+    {
+      "epoch": 0.7169042769857433,
+      "grad_norm": 1.0390625,
+      "learning_rate": 9.517455302940388e-06,
+      "loss": 1.098876953125,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 3.00079,
+      "step": 22,
+      "tokens/total": 5767168,
+      "tokens/train_per_sec_per_gpu": 255.12,
+      "tokens/trainable": 5754904
+    },
+    {
+      "epoch": 0.7494908350305499,
+      "grad_norm": 1.3828125,
+      "learning_rate": 9.46347054878559e-06,
+      "loss": 1.1710205078125,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 3.22528,
+      "step": 23,
+      "tokens/total": 6029312,
+      "tokens/train_per_sec_per_gpu": 255.11,
+      "tokens/trainable": 6016307
+    },
+    {
+      "epoch": 0.7820773930753564,
+      "grad_norm": 1.171875,
+      "learning_rate": 9.406814140443898e-06,
+      "loss": 1.0035400390625,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 2.72792,
+      "step": 24,
+      "tokens/total": 6291456,
+      "tokens/train_per_sec_per_gpu": 255.74,
+      "tokens/trainable": 6277895
+    },
+    {
+      "epoch": 0.814663951120163,
+      "grad_norm": 1.578125,
+      "learning_rate": 9.347524268266092e-06,
+      "loss": 1.2330322265625,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 3.43162,
+      "step": 25,
+      "tokens/total": 6553600,
+      "tokens/train_per_sec_per_gpu": 254.29,
+      "tokens/trainable": 6539342
+    },
+    {
+      "epoch": 0.8472505091649695,
+      "grad_norm": 1.0390625,
+      "learning_rate": 9.285640897740316e-06,
+      "loss": 1.1314697265625,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 3.10021,
+      "step": 26,
+      "tokens/total": 6815744,
+      "tokens/train_per_sec_per_gpu": 254.73,
+      "tokens/trainable": 6800934
+    },
+    {
+      "epoch": 0.879837067209776,
+      "grad_norm": 1.8046875,
+      "learning_rate": 9.22120574255258e-06,
+      "loss": 1.0982666015625,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 2.99896,
+      "step": 27,
+      "tokens/total": 7077888,
+      "tokens/train_per_sec_per_gpu": 255.33,
+      "tokens/trainable": 7062352
+    },
+    {
+      "epoch": 0.9124236252545825,
+      "grad_norm": 1.265625,
+      "learning_rate": 9.154262236468826e-06,
+      "loss": 1.099609375,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 3.00299,
+      "step": 28,
+      "tokens/total": 7340032,
+      "tokens/train_per_sec_per_gpu": 253.72,
+      "tokens/trainable": 7323488
+    },
+    {
+      "epoch": 0.945010183299389,
+      "grad_norm": 1.1875,
+      "learning_rate": 9.084855504057562e-06,
+      "loss": 1.1102294921875,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 3.03505,
+      "step": 29,
+      "tokens/total": 7602176,
+      "tokens/train_per_sec_per_gpu": 256.49,
+      "tokens/trainable": 7584813
+    },
+    {
+      "epoch": 0.9775967413441955,
+      "grad_norm": 1.3984375,
+      "learning_rate": 9.013032330272777e-06,
+      "loss": 1.1253662109375,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 3.08135,
+      "step": 30,
+      "tokens/total": 7864320,
+      "tokens/train_per_sec_per_gpu": 254.62,
+      "tokens/trainable": 7846058
+    },
+    {
+      "epoch": 1.0,
+      "grad_norm": 2.1875,
+      "learning_rate": 8.938841128917622e-06,
+      "loss": 1.208984375,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 3.35008,
+      "step": 31,
+      "tokens/total": 8044544,
+      "tokens/train_per_sec_per_gpu": 189.49,
+      "tokens/trainable": 8017124
+    },
+    {
+      "epoch": 1.0325865580448066,
+      "grad_norm": 4.71875,
+      "learning_rate": 8.86233191001016e-06,
+      "loss": 1.2061767578125,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 3.34069,
+      "step": 32,
+      "tokens/total": 8306688,
+      "tokens/train_per_sec_per_gpu": 253.6,
+      "tokens/trainable": 8278814
+    },
+    {
+      "epoch": 1.065173116089613,
+      "grad_norm": 2.078125,
+      "learning_rate": 8.783556246073135e-06,
+      "loss": 1.1258544921875,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 3.08285,
+      "step": 33,
+      "tokens/total": 8568832,
+      "tokens/train_per_sec_per_gpu": 254.33,
+      "tokens/trainable": 8540499
+    },
+    {
+      "epoch": 1.0977596741344195,
+      "grad_norm": 1.9140625,
+      "learning_rate": 8.702567237370521e-06,
+      "loss": 1.1329345703125,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 3.10475,
+      "step": 34,
+      "tokens/total": 8830976,
+      "tokens/train_per_sec_per_gpu": 255.14,
+      "tokens/trainable": 8802205
+    },
+    {
+      "epoch": 1.1303462321792261,
+      "grad_norm": 1.046875,
+      "learning_rate": 8.619419476114251e-06,
+      "loss": 1.035400390625,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 2.81623,
+      "step": 35,
+      "tokens/total": 9093120,
+      "tokens/train_per_sec_per_gpu": 255.14,
+      "tokens/trainable": 9063740
+    },
+    {
+      "epoch": 1.1629327902240325,
+      "grad_norm": 1.15625,
+      "learning_rate": 8.534169009665282e-06,
+      "loss": 1.0863037109375,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 2.9633,
+      "step": 36,
+      "tokens/total": 9355264,
+      "tokens/train_per_sec_per_gpu": 254.09,
+      "tokens/trainable": 9325182
+    },
+    {
+      "epoch": 1.195519348268839,
+      "grad_norm": 1.125,
+      "learning_rate": 8.446873302753783e-06,
+      "loss": 1.094482421875,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 2.98764,
+      "step": 37,
+      "tokens/total": 9617408,
+      "tokens/train_per_sec_per_gpu": 254.54,
+      "tokens/trainable": 9586714
+    },
+    {
+      "epoch": 1.2281059063136457,
+      "grad_norm": 2.171875,
+      "learning_rate": 8.357591198743923e-06,
+      "loss": 1.06298828125,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 2.89501,
+      "step": 38,
+      "tokens/total": 9879552,
+      "tokens/train_per_sec_per_gpu": 254.93,
+      "tokens/trainable": 9848284
+    },
+    {
+      "epoch": 1.260692464358452,
+      "grad_norm": 1.6015625,
+      "learning_rate": 8.266382879969356e-06,
+      "loss": 1.0103759765625,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 2.74663,
+      "step": 39,
+      "tokens/total": 10141696,
+      "tokens/train_per_sec_per_gpu": 255.95,
+      "tokens/trainable": 10109921
+    },
+    {
+      "epoch": 1.2932790224032586,
+      "grad_norm": 1.375,
+      "learning_rate": 8.17330982716615e-06,
+      "loss": 1.096923828125,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 2.99494,
+      "step": 40,
+      "tokens/total": 10403840,
+      "tokens/train_per_sec_per_gpu": 250.18,
+      "tokens/trainable": 10371483
+    },
+    {
+      "epoch": 1.3258655804480652,
+      "grad_norm": 2.328125,
+      "learning_rate": 8.078434778030511e-06,
+      "loss": 1.08544921875,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 2.96077,
+      "step": 41,
+      "tokens/total": 10665984,
+      "tokens/train_per_sec_per_gpu": 254.08,
+      "tokens/trainable": 10632994
+    },
+    {
+      "epoch": 1.3584521384928716,
+      "grad_norm": 4.625,
+      "learning_rate": 7.981821684929218e-06,
+      "loss": 1.2464599609375,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 3.47801,
+      "step": 42,
+      "tokens/total": 10928128,
+      "tokens/train_per_sec_per_gpu": 254.42,
+      "tokens/trainable": 10894583
+    },
+    {
+      "epoch": 1.3910386965376782,
+      "grad_norm": 1.296875,
+      "learning_rate": 7.883535671791294e-06,
+      "loss": 1.100341796875,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 3.00519,
+      "step": 43,
+      "tokens/total": 11190272,
+      "tokens/train_per_sec_per_gpu": 254.37,
+      "tokens/trainable": 11156144
+    },
+    {
+      "epoch": 1.4236252545824848,
+      "grad_norm": 2.71875,
+      "learning_rate": 7.783642990209951e-06,
+      "loss": 0.9239501953125,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 2.51922,
+      "step": 44,
+      "tokens/total": 11452416,
+      "tokens/train_per_sec_per_gpu": 255.65,
+      "tokens/trainable": 11417657
+    },
+    {
+      "epoch": 1.4562118126272914,
+      "grad_norm": 0.984375,
+      "learning_rate": 7.682210974784426e-06,
+      "loss": 1.04559326171875,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 2.84509,
+      "step": 45,
+      "tokens/total": 11714560,
+      "tokens/train_per_sec_per_gpu": 253.21,
+      "tokens/trainable": 11679451
+    },
+    {
+      "epoch": 1.4887983706720977,
+      "grad_norm": 1.4921875,
+      "learning_rate": 7.579307997731783e-06,
+      "loss": 1.1317138671875,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 3.10097,
+      "step": 46,
+      "tokens/total": 11976704,
+      "tokens/train_per_sec_per_gpu": 253.66,
+      "tokens/trainable": 11941075
+    },
+    {
+      "epoch": 1.5213849287169041,
+      "grad_norm": 1.078125,
+      "learning_rate": 7.475003422799302e-06,
+      "loss": 1.0257568359375,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 2.78921,
+      "step": 47,
+      "tokens/total": 12238848,
+      "tokens/train_per_sec_per_gpu": 255.11,
+      "tokens/trainable": 12202639
+    },
+    {
+      "epoch": 1.5539714867617107,
+      "grad_norm": 1.5625,
+      "learning_rate": 7.36936755850849e-06,
+      "loss": 1.0682373046875,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 2.91025,
+      "step": 48,
+      "tokens/total": 12500992,
+      "tokens/train_per_sec_per_gpu": 252.51,
+      "tokens/trainable": 12464200
+    },
+    {
+      "epoch": 1.5865580448065173,
+      "grad_norm": 1.265625,
+      "learning_rate": 7.2624716107622675e-06,
+      "loss": 1.0413818359375,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 2.83313,
+      "step": 49,
+      "tokens/total": 12763136,
+      "tokens/train_per_sec_per_gpu": 255.44,
+      "tokens/trainable": 12725850
+    },
+    {
+      "epoch": 1.6191446028513239,
+      "grad_norm": 1.40625,
+      "learning_rate": 7.154387634847241e-06,
+      "loss": 1.0535888671875,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 2.86793,
+      "step": 50,
+      "tokens/total": 13025280,
+      "tokens/train_per_sec_per_gpu": 255.34,
+      "tokens/trainable": 12987575
+    },
+    {
+      "epoch": 1.6517311608961305,
+      "grad_norm": 1.1953125,
+      "learning_rate": 7.045188486863449e-06,
+      "loss": 1.037353515625,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 2.82174,
+      "step": 51,
+      "tokens/total": 13287424,
+      "tokens/train_per_sec_per_gpu": 254.48,
+      "tokens/trainable": 13249123
+    },
+    {
+      "epoch": 1.6843177189409368,
+      "grad_norm": 1.703125,
+      "learning_rate": 6.9349477746142846e-06,
+      "loss": 0.979248046875,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 2.66245,
+      "step": 52,
+      "tokens/total": 13549568,
+      "tokens/train_per_sec_per_gpu": 255.94,
+      "tokens/trainable": 13510636
+    },
+    {
+      "epoch": 1.7169042769857432,
+      "grad_norm": 0.98828125,
+      "learning_rate": 6.823739807989734e-06,
+      "loss": 0.972900390625,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 2.64561,
+      "step": 53,
+      "tokens/total": 13811712,
+      "tokens/train_per_sec_per_gpu": 255.27,
+      "tokens/trainable": 13772028
+    },
+    {
+      "epoch": 1.7494908350305498,
+      "grad_norm": 1.0,
+      "learning_rate": 6.7116395488763565e-06,
+      "loss": 1.0552978515625,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 2.87283,
+      "step": 54,
+      "tokens/total": 14073856,
+      "tokens/train_per_sec_per_gpu": 255.32,
+      "tokens/trainable": 14033431
+    },
+    {
+      "epoch": 1.7820773930753564,
+      "grad_norm": 1.3203125,
+      "learning_rate": 6.598722560627761e-06,
+      "loss": 0.8934326171875,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 2.4435,
+      "step": 55,
+      "tokens/total": 14336000,
+      "tokens/train_per_sec_per_gpu": 256.15,
+      "tokens/trainable": 14295019
+    },
+    {
+      "epoch": 1.814663951120163,
+      "grad_norm": 1.125,
+      "learning_rate": 6.485064957129677e-06,
+      "loss": 1.1385498046875,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 3.12224,
+      "step": 56,
+      "tokens/total": 14598144,
+      "tokens/train_per_sec_per_gpu": 254.38,
+      "tokens/trainable": 14556466
+    },
+    {
+      "epoch": 1.8472505091649696,
+      "grad_norm": 1.15625,
+      "learning_rate": 6.370743351493899e-06,
+      "loss": 1.02783203125,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 2.795,
+      "step": 57,
+      "tokens/total": 14860288,
+      "tokens/train_per_sec_per_gpu": 254.73,
+      "tokens/trainable": 14818058
+    },
+    {
+      "epoch": 1.879837067209776,
+      "grad_norm": 4.125,
+      "learning_rate": 6.255834804415742e-06,
+      "loss": 1.0128173828125,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 2.75335,
+      "step": 58,
+      "tokens/total": 15122432,
+      "tokens/train_per_sec_per_gpu": 252.83,
+      "tokens/trainable": 15079476
+    },
+    {
+      "epoch": 1.9124236252545825,
+      "grad_norm": 1.2890625,
+      "learning_rate": 6.140416772229785e-06,
+      "loss": 1.0045166015625,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 2.73059,
+      "step": 59,
+      "tokens/total": 15384576,
+      "tokens/train_per_sec_per_gpu": 253.36,
+      "tokens/trainable": 15340612
+    },
+    {
+      "epoch": 1.945010183299389,
+      "grad_norm": 1.1015625,
+      "learning_rate": 6.0245670546989165e-06,
+      "loss": 1.01708984375,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 2.76514,
+      "step": 60,
+      "tokens/total": 15646720,
+      "tokens/train_per_sec_per_gpu": 256.27,
+      "tokens/trainable": 15601937
+    },
+    {
+      "epoch": 1.9775967413441955,
+      "grad_norm": 1.0859375,
+      "learning_rate": 5.908363742571915e-06,
+      "loss": 1.04248046875,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 2.83624,
+      "step": 61,
+      "tokens/total": 15908864,
+      "tokens/train_per_sec_per_gpu": 254.31,
+      "tokens/trainable": 15863182
+    },
+    {
+      "epoch": 2.0,
+      "grad_norm": 1.3125,
+      "learning_rate": 5.791885164944844e-06,
+      "loss": 1.118896484375,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 3.06147,
+      "step": 62,
+      "tokens/total": 16089088,
+      "tokens/train_per_sec_per_gpu": 190.14,
+      "tokens/trainable": 16034248
+    },
+    {
+      "epoch": 2.0325865580448066,
+      "grad_norm": 1.3515625,
+      "learning_rate": 5.67520983646182e-06,
+      "loss": 1.15771484375,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 3.18265,
+      "step": 63,
+      "tokens/total": 16351232,
+      "tokens/train_per_sec_per_gpu": 253.6,
+      "tokens/trainable": 16295938
+    },
+    {
+      "epoch": 2.065173116089613,
+      "grad_norm": 1.0703125,
+      "learning_rate": 5.5584164043906895e-06,
+      "loss": 1.06201171875,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 2.89218,
+      "step": 64,
+      "tokens/total": 16613376,
+      "tokens/train_per_sec_per_gpu": 254.19,
+      "tokens/trainable": 16557623
+    },
+    {
+      "epoch": 2.0977596741344193,
+      "grad_norm": 1.015625,
+      "learning_rate": 5.441583595609312e-06,
+      "loss": 1.06689453125,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 2.90634,
+      "step": 65,
+      "tokens/total": 16875520,
+      "tokens/train_per_sec_per_gpu": 255.11,
+      "tokens/trainable": 16819328
+    },
+    {
+      "epoch": 2.130346232179226,
+      "grad_norm": 1.28125,
+      "learning_rate": 5.324790163538181e-06,
+      "loss": 0.959716796875,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 2.61096,
+      "step": 66,
+      "tokens/total": 17137664,
+      "tokens/train_per_sec_per_gpu": 255.19,
+      "tokens/trainable": 17080864
+    },
+    {
+      "epoch": 2.1629327902240325,
+      "grad_norm": 0.99609375,
+      "learning_rate": 5.208114835055157e-06,
+      "loss": 1.015869140625,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 2.76176,
+      "step": 67,
+      "tokens/total": 17399808,
+      "tokens/train_per_sec_per_gpu": 254.37,
+      "tokens/trainable": 17342308
+    },
+    {
+      "epoch": 2.195519348268839,
+      "grad_norm": 1.25,
+      "learning_rate": 5.0916362574280864e-06,
+      "loss": 1.029541015625,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 2.79978,
+      "step": 68,
+      "tokens/total": 17661952,
+      "tokens/train_per_sec_per_gpu": 254.67,
+      "tokens/trainable": 17603842
+    },
+    {
+      "epoch": 2.2281059063136457,
+      "grad_norm": 2.0625,
+      "learning_rate": 4.975432945301085e-06,
+      "loss": 1.0140380859375,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 2.75671,
+      "step": 69,
+      "tokens/total": 17924096,
+      "tokens/train_per_sec_per_gpu": 254.95,
+      "tokens/trainable": 17865408
+    },
+    {
+      "epoch": 2.2606924643584523,
+      "grad_norm": 1.015625,
+      "learning_rate": 4.859583227770218e-06,
+      "loss": 0.9541015625,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 2.59634,
+      "step": 70,
+      "tokens/total": 18186240,
+      "tokens/train_per_sec_per_gpu": 256.18,
+      "tokens/trainable": 18127044
+    },
+    {
+      "epoch": 2.293279022403259,
+      "grad_norm": 1.2421875,
+      "learning_rate": 4.744165195584258e-06,
+      "loss": 1.041259765625,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 2.83278,
+      "step": 71,
+      "tokens/total": 18448384,
+      "tokens/train_per_sec_per_gpu": 255.1,
+      "tokens/trainable": 18388602
+    },
+    {
+      "epoch": 2.325865580448065,
+      "grad_norm": 1.3984375,
+      "learning_rate": 4.6292566485061015e-06,
+      "loss": 1.0445556640625,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 2.84214,
+      "step": 72,
+      "tokens/total": 18710528,
+      "tokens/train_per_sec_per_gpu": 254.16,
+      "tokens/trainable": 18650108
+    },
+    {
+      "epoch": 2.3584521384928716,
+      "grad_norm": 1.1484375,
+      "learning_rate": 4.514935042870324e-06,
+      "loss": 1.213623046875,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 3.36566,
+      "step": 73,
+      "tokens/total": 18972672,
+      "tokens/train_per_sec_per_gpu": 254.87,
+      "tokens/trainable": 18911704
+    },
+    {
+      "epoch": 2.391038696537678,
+      "grad_norm": 1.234375,
+      "learning_rate": 4.40127743937224e-06,
+      "loss": 1.045166015625,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 2.84387,
+      "step": 74,
+      "tokens/total": 19234816,
+      "tokens/train_per_sec_per_gpu": 254.42,
+      "tokens/trainable": 19173264
+    },
+    {
+      "epoch": 2.423625254582485,
+      "grad_norm": 2.25,
+      "learning_rate": 4.288360451123646e-06,
+      "loss": 0.8837890625,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 2.42005,
+      "step": 75,
+      "tokens/total": 19496960,
+      "tokens/train_per_sec_per_gpu": 255.7,
+      "tokens/trainable": 19434780
+    },
+    {
+      "epoch": 2.4562118126272914,
+      "grad_norm": 2.859375,
+      "learning_rate": 4.1762601920102675e-06,
+      "loss": 0.9906005859375,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 2.69285,
+      "step": 76,
+      "tokens/total": 19759104,
+      "tokens/train_per_sec_per_gpu": 253.15,
+      "tokens/trainable": 19696572
+    },
+    {
+      "epoch": 2.4887983706720975,
+      "grad_norm": 1.1171875,
+      "learning_rate": 4.065052225385717e-06,
+      "loss": 1.084716796875,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 2.9586,
+      "step": 77,
+      "tokens/total": 20021248,
+      "tokens/train_per_sec_per_gpu": 253.49,
+      "tokens/trainable": 19958192
+    },
+    {
+      "epoch": 2.521384928716904,
+      "grad_norm": 1.2890625,
+      "learning_rate": 3.954811513136554e-06,
+      "loss": 0.9735107421875,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 2.64722,
+      "step": 78,
+      "tokens/total": 20283392,
+      "tokens/train_per_sec_per_gpu": 255.53,
+      "tokens/trainable": 20219756
+    },
+    {
+      "epoch": 2.5539714867617107,
+      "grad_norm": 1.2734375,
+      "learning_rate": 3.84561236515276e-06,
+      "loss": 1.0286865234375,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 2.79739,
+      "step": 79,
+      "tokens/total": 20545536,
+      "tokens/train_per_sec_per_gpu": 252.65,
+      "tokens/trainable": 20481318
+    },
+    {
+      "epoch": 2.5865580448065173,
+      "grad_norm": 1.171875,
+      "learning_rate": 3.7375283892377344e-06,
+      "loss": 0.99755859375,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 2.71165,
+      "step": 80,
+      "tokens/total": 20807680,
+      "tokens/train_per_sec_per_gpu": 255.68,
+      "tokens/trainable": 20742968
+    },
+    {
+      "epoch": 2.619144602851324,
+      "grad_norm": 1.6015625,
+      "learning_rate": 3.630632441491512e-06,
+      "loss": 1.01611328125,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 2.76244,
+      "step": 81,
+      "tokens/total": 21069824,
+      "tokens/train_per_sec_per_gpu": 255.54,
+      "tokens/trainable": 21004696
+    },
+    {
+      "epoch": 2.6517311608961305,
+      "grad_norm": 1.0546875,
+      "learning_rate": 3.5249965772007e-06,
+      "loss": 0.997314453125,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 2.71099,
+      "step": 82,
+      "tokens/total": 21331968,
+      "tokens/train_per_sec_per_gpu": 254.35,
+      "tokens/trainable": 21266244
+    },
+    {
+      "epoch": 2.684317718940937,
+      "grad_norm": 1.3671875,
+      "learning_rate": 3.4206920022682173e-06,
+      "loss": 0.9415283203125,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 2.5639,
+      "step": 83,
+      "tokens/total": 21594112,
+      "tokens/train_per_sec_per_gpu": 255.48,
+      "tokens/trainable": 21527762
+    },
+    {
+      "epoch": 2.716904276985743,
+      "grad_norm": 1.015625,
+      "learning_rate": 3.3177890252155755e-06,
+      "loss": 0.9327392578125,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 2.54146,
+      "step": 84,
+      "tokens/total": 21856256,
+      "tokens/train_per_sec_per_gpu": 255.1,
+      "tokens/trainable": 21789156
+    },
+    {
+      "epoch": 2.74949083503055,
+      "grad_norm": 2.71875,
+      "learning_rate": 3.2163570097900497e-06,
+      "loss": 1.014404296875,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 2.75772,
+      "step": 85,
+      "tokens/total": 22118400,
+      "tokens/train_per_sec_per_gpu": 255.17,
+      "tokens/trainable": 22050560
+    },
+    {
+      "epoch": 2.7820773930753564,
+      "grad_norm": 1.0625,
+      "learning_rate": 3.116464328208708e-06,
+      "loss": 0.8604736328125,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 2.36428,
+      "step": 86,
+      "tokens/total": 22380544,
+      "tokens/train_per_sec_per_gpu": 255.95,
+      "tokens/trainable": 22312150
+    },
+    {
+      "epoch": 2.814663951120163,
+      "grad_norm": 1.359375,
+      "learning_rate": 3.0181783150707827e-06,
+      "loss": 1.10498046875,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 3.01917,
+      "step": 87,
+      "tokens/total": 22642688,
+      "tokens/train_per_sec_per_gpu": 254.3,
+      "tokens/trainable": 22573596
+    },
+    {
+      "epoch": 2.8472505091649696,
+      "grad_norm": 2.1875,
+      "learning_rate": 2.921565221969492e-06,
+      "loss": 0.997802734375,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 2.71232,
+      "step": 88,
+      "tokens/total": 22904832,
+      "tokens/train_per_sec_per_gpu": 254.74,
+      "tokens/trainable": 22835190
+    },
+    {
+      "epoch": 2.8798370672097757,
+      "grad_norm": 1.7421875,
+      "learning_rate": 2.8266901728338526e-06,
+      "loss": 0.99560546875,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 2.70636,
+      "step": 89,
+      "tokens/total": 23166976,
+      "tokens/train_per_sec_per_gpu": 255.35,
+      "tokens/trainable": 23096608
+    },
+    {
+      "epoch": 2.9124236252545828,
+      "grad_norm": 1.3828125,
+      "learning_rate": 2.7336171200306467e-06,
+      "loss": 0.9744873046875,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 2.64981,
+      "step": 90,
+      "tokens/total": 23429120,
+      "tokens/train_per_sec_per_gpu": 253.57,
+      "tokens/trainable": 23357748
+    },
+    {
+      "epoch": 2.945010183299389,
+      "grad_norm": 1.546875,
+      "learning_rate": 2.6424088012560766e-06,
+      "loss": 0.9864501953125,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 2.6817,
+      "step": 91,
+      "tokens/total": 23691264,
+      "tokens/train_per_sec_per_gpu": 256.57,
+      "tokens/trainable": 23619068
+    },
+    {
+      "epoch": 2.9775967413441955,
+      "grad_norm": 0.953125,
+      "learning_rate": 2.5531266972462176e-06,
+      "loss": 1.0126953125,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 2.75301,
+      "step": 92,
+      "tokens/total": 23953408,
+      "tokens/train_per_sec_per_gpu": 254.73,
+      "tokens/trainable": 23880316
+    },
+    {
+      "epoch": 3.0,
+      "grad_norm": 1.34375,
+      "learning_rate": 2.4658309903347196e-06,
+      "loss": 1.079345703125,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 2.94275,
+      "step": 93,
+      "tokens/total": 24133632,
+      "tokens/train_per_sec_per_gpu": 190.27,
+      "tokens/trainable": 24051380
+    },
+    {
+      "epoch": 3.0325865580448066,
+      "grad_norm": 1.375,
+      "learning_rate": 2.380580523885751e-06,
+      "loss": 1.1339111328125,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 3.10779,
+      "step": 94,
+      "tokens/total": 24395776,
+      "tokens/train_per_sec_per_gpu": 245.35,
+      "tokens/trainable": 24313072
+    },
+    {
+      "epoch": 3.065173116089613,
+      "grad_norm": 2.84375,
+      "learning_rate": 2.29743276262948e-06,
+      "loss": 1.0352783203125,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 2.81589,
+      "step": 95,
+      "tokens/total": 24657920,
+      "tokens/train_per_sec_per_gpu": 251.51,
+      "tokens/trainable": 24574758
+    },
+    {
+      "epoch": 3.0977596741344193,
+      "grad_norm": 1.2109375,
+      "learning_rate": 2.2164437539268652e-06,
+      "loss": 1.0400390625,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 2.82933,
+      "step": 96,
+      "tokens/total": 24920064,
+      "tokens/train_per_sec_per_gpu": 255.22,
+      "tokens/trainable": 24836468
+    },
+    {
+      "epoch": 3.130346232179226,
+      "grad_norm": 0.96875,
+      "learning_rate": 2.1376680899898415e-06,
+      "loss": 0.9400634765625,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 2.56014,
+      "step": 97,
+      "tokens/total": 25182208,
+      "tokens/train_per_sec_per_gpu": 255.18,
+      "tokens/trainable": 25098004
+    },
+    {
+      "epoch": 3.1629327902240325,
+      "grad_norm": 0.859375,
+      "learning_rate": 2.0611588710823797e-06,
+      "loss": 0.994140625,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 2.7024,
+      "step": 98,
+      "tokens/total": 25444352,
+      "tokens/train_per_sec_per_gpu": 254.29,
+      "tokens/trainable": 25359448
+    },
+    {
+      "epoch": 3.195519348268839,
+      "grad_norm": 1.046875,
+      "learning_rate": 1.986967669727224e-06,
+      "loss": 1.01171875,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 2.75032,
+      "step": 99,
+      "tokens/total": 25706496,
+      "tokens/train_per_sec_per_gpu": 254.37,
+      "tokens/trainable": 25620982
+    },
+    {
+      "epoch": 3.2281059063136457,
+      "grad_norm": 1.421875,
+      "learning_rate": 1.9151444959424383e-06,
+      "loss": 0.9998779296875,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 2.71795,
+      "step": 100,
+      "tokens/total": 25968640,
+      "tokens/train_per_sec_per_gpu": 254.65,
+      "tokens/trainable": 25882548
+    },
+    {
+      "epoch": 3.2606924643584523,
+      "grad_norm": 3.046875,
+      "learning_rate": 1.8457377635311763e-06,
+      "loss": 0.9356689453125,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 2.54892,
+      "step": 101,
+      "tokens/total": 26230784,
+      "tokens/train_per_sec_per_gpu": 255.75,
+      "tokens/trainable": 26144184
+    },
+    {
+      "epoch": 3.293279022403259,
+      "grad_norm": 0.93359375,
+      "learning_rate": 1.7787942574474215e-06,
+      "loss": 1.0257568359375,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 2.78921,
+      "step": 102,
+      "tokens/total": 26492928,
+      "tokens/train_per_sec_per_gpu": 255.11,
+      "tokens/trainable": 26405742
+    },
+    {
+      "epoch": 3.325865580448065,
+      "grad_norm": 1.4453125,
+      "learning_rate": 1.7143591022596846e-06,
+      "loss": 1.0308837890625,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 2.80354,
+      "step": 103,
+      "tokens/total": 26755072,
+      "tokens/train_per_sec_per_gpu": 253.94,
+      "tokens/trainable": 26667248
+    },
+    {
+      "epoch": 3.3584521384928716,
+      "grad_norm": 0.984375,
+      "learning_rate": 1.6524757317339102e-06,
+      "loss": 1.19873046875,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 3.3159,
+      "step": 104,
+      "tokens/total": 27017216,
+      "tokens/train_per_sec_per_gpu": 254.4,
+      "tokens/trainable": 26928844
+    },
+    {
+      "epoch": 3.391038696537678,
+      "grad_norm": 1.03125,
+      "learning_rate": 1.593185859556103e-06,
+      "loss": 1.0316162109375,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 2.8056,
+      "step": 105,
+      "tokens/total": 27279360,
+      "tokens/train_per_sec_per_gpu": 254.46,
+      "tokens/trainable": 27190404
+    },
+    {
+      "epoch": 3.423625254582485,
+      "grad_norm": 1.5234375,
+      "learning_rate": 1.5365294512144114e-06,
+      "loss": 0.8751220703125,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 2.39917,
+      "step": 106,
+      "tokens/total": 27541504,
+      "tokens/train_per_sec_per_gpu": 255.65,
+      "tokens/trainable": 27451920
+    },
+    {
+      "epoch": 3.4562118126272914,
+      "grad_norm": 0.890625,
+      "learning_rate": 1.4825446970596136e-06,
+      "loss": 0.9832763671875,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 2.6732,
+      "step": 107,
+      "tokens/total": 27803648,
+      "tokens/train_per_sec_per_gpu": 253.39,
+      "tokens/trainable": 27713712
+    },
+    {
+      "epoch": 3.4887983706720975,
+      "grad_norm": 2.8125,
+      "learning_rate": 1.4312679865621742e-06,
+      "loss": 1.0726318359375,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 2.92306,
+      "step": 108,
+      "tokens/total": 28065792,
+      "tokens/train_per_sec_per_gpu": 253.75,
+      "tokens/trainable": 27975332
+    },
+    {
+      "epoch": 3.521384928716904,
+      "grad_norm": 0.921875,
+      "learning_rate": 1.382733883783211e-06,
+      "loss": 0.9619140625,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 2.6167,
+      "step": 109,
+      "tokens/total": 28327936,
+      "tokens/train_per_sec_per_gpu": 255.24,
+      "tokens/trainable": 28236896
+    },
+    {
+      "epoch": 3.5539714867617107,
+      "grad_norm": 1.0,
+      "learning_rate": 1.3369751040759236e-06,
+      "loss": 1.0186767578125,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 2.76953,
+      "step": 110,
+      "tokens/total": 28590080,
+      "tokens/train_per_sec_per_gpu": 252.49,
+      "tokens/trainable": 28498458
+    },
+    {
+      "epoch": 3.5865580448065173,
+      "grad_norm": 1.0546875,
+      "learning_rate": 1.2940224920331707e-06,
+      "loss": 0.98779296875,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 2.6853,
+      "step": 111,
+      "tokens/total": 28852224,
+      "tokens/train_per_sec_per_gpu": 255.71,
+      "tokens/trainable": 28760108
+    },
+    {
+      "epoch": 3.619144602851324,
+      "grad_norm": 0.953125,
+      "learning_rate": 1.2539050006960814e-06,
+      "loss": 1.0091552734375,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 2.74328,
+      "step": 112,
+      "tokens/total": 29114368,
+      "tokens/train_per_sec_per_gpu": 255.57,
+      "tokens/trainable": 29021836
+    },
+    {
+      "epoch": 3.6517311608961305,
+      "grad_norm": 0.9765625,
+      "learning_rate": 1.2166496720376874e-06,
+      "loss": 0.988525390625,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 2.68727,
+      "step": 113,
+      "tokens/total": 29376512,
+      "tokens/train_per_sec_per_gpu": 252.61,
+      "tokens/trainable": 29283384
+    },
+    {
+      "epoch": 3.684317718940937,
+      "grad_norm": 7.53125,
+      "learning_rate": 1.1822816187347625e-06,
+      "loss": 0.933837890625,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 2.54426,
+      "step": 114,
+      "tokens/total": 29638656,
+      "tokens/train_per_sec_per_gpu": 253.68,
+      "tokens/trainable": 29544902
+    },
+    {
+      "epoch": 3.716904276985743,
+      "grad_norm": 0.8984375,
+      "learning_rate": 1.1508240072401336e-06,
+      "loss": 0.925048828125,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 2.52199,
+      "step": 115,
+      "tokens/total": 29900800,
+      "tokens/train_per_sec_per_gpu": 255.09,
+      "tokens/trainable": 29806296
+    },
+    {
+      "epoch": 3.74949083503055,
+      "grad_norm": 0.90234375,
+      "learning_rate": 1.1222980421668874e-06,
+      "loss": 1.0098876953125,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 2.74529,
+      "step": 116,
+      "tokens/total": 30162944,
+      "tokens/train_per_sec_per_gpu": 255.13,
+      "tokens/trainable": 30067700
+    },
+    {
+      "epoch": 3.7820773930753564,
+      "grad_norm": 1.0,
+      "learning_rate": 1.0967229519949833e-06,
+      "loss": 0.8541259765625,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 2.34932,
+      "step": 117,
+      "tokens/total": 30425088,
+      "tokens/train_per_sec_per_gpu": 256.06,
+      "tokens/trainable": 30329290
+    },
+    {
+      "epoch": 3.814663951120163,
+      "grad_norm": 1.09375,
+      "learning_rate": 1.0741159761099294e-06,
+      "loss": 1.100341796875,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 3.00519,
+      "step": 118,
+      "tokens/total": 30687232,
+      "tokens/train_per_sec_per_gpu": 254.3,
+      "tokens/trainable": 30590736
+    },
+    {
+      "epoch": 3.8472505091649696,
+      "grad_norm": 1.03125,
+      "learning_rate": 1.054492353182237e-06,
+      "loss": 0.9945068359375,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 2.70339,
+      "step": 119,
+      "tokens/total": 30949376,
+      "tokens/train_per_sec_per_gpu": 254.64,
+      "tokens/trainable": 30852330
+    },
+    {
+      "epoch": 3.8798370672097757,
+      "grad_norm": 1.859375,
+      "learning_rate": 1.0378653108955017e-06,
+      "loss": 0.9912109375,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 2.6945,
+      "step": 120,
+      "tokens/total": 31211520,
+      "tokens/train_per_sec_per_gpu": 255.23,
+      "tokens/trainable": 31113748
+    },
+    {
+      "epoch": 3.9124236252545828,
+      "grad_norm": 0.859375,
+      "learning_rate": 1.0242460570300241e-06,
+      "loss": 0.9700927734375,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 2.63819,
+      "step": 121,
+      "tokens/total": 31473664,
+      "tokens/train_per_sec_per_gpu": 253.4,
+      "tokens/trainable": 31374888
+    },
+    {
+      "epoch": 3.945010183299389,
+      "grad_norm": 0.9921875,
+      "learning_rate": 1.01364377190799e-06,
+      "loss": 0.9827880859375,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 2.6719,
+      "step": 122,
+      "tokens/total": 31735808,
+      "tokens/train_per_sec_per_gpu": 256.6,
+      "tokens/trainable": 31636208
+    },
+    {
+      "epoch": 3.9775967413441955,
+      "grad_norm": 4.0625,
+      "learning_rate": 1.0060656022052966e-06,
+      "loss": 1.007080078125,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 2.7376,
+      "step": 123,
+      "tokens/total": 31997952,
+      "tokens/train_per_sec_per_gpu": 254.5,
+      "tokens/trainable": 31897456
+    },
+    {
+      "epoch": 4.0,
+      "grad_norm": 1.390625,
+      "learning_rate": 1.0015166561341943e-06,
+      "loss": 1.0748291015625,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 2.92949,
+      "step": 124,
+      "tokens/total": 32178176,
+      "tokens/train_per_sec_per_gpu": 190.13,
+      "tokens/trainable": 32068520
+    }
+  ],
+  "logging_steps": 1,
+  "max_steps": 124,
+  "num_input_tokens_seen": 0,
+  "num_train_epochs": 4,
+  "save_steps": 500,
+  "stateful_callbacks": {
+    "TrainerControl": {
+      "args": {
+        "should_epoch_stop": false,
+        "should_evaluate": false,
+        "should_log": false,
+        "should_save": true,
+        "should_training_stop": true
+      },
+      "attributes": {}
+    }
+  },
+  "total_flos": 2.157687214690009e+18,
+  "train_batch_size": 1,
+  "trial_name": null,
+  "trial_params": null
+}