Instructions to use jbostock/scimt-dispatch-midtrained-sft-v1 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use jbostock/scimt-dispatch-midtrained-sft-v1 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="jbostock/scimt-dispatch-midtrained-sft-v1")

# Load model directly
from transformers import AutoModel
model = AutoModel.from_pretrained("jbostock/scimt-dispatch-midtrained-sft-v1", device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use jbostock/scimt-dispatch-midtrained-sft-v1 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "jbostock/scimt-dispatch-midtrained-sft-v1"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "jbostock/scimt-dispatch-midtrained-sft-v1",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/jbostock/scimt-dispatch-midtrained-sft-v1

SGLang

How to use jbostock/scimt-dispatch-midtrained-sft-v1 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "jbostock/scimt-dispatch-midtrained-sft-v1" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "jbostock/scimt-dispatch-midtrained-sft-v1",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "jbostock/scimt-dispatch-midtrained-sft-v1" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "jbostock/scimt-dispatch-midtrained-sft-v1",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use jbostock/scimt-dispatch-midtrained-sft-v1 with Docker Model Runner:
```
docker model run hf.co/jbostock/scimt-dispatch-midtrained-sft-v1
```

jbostock commited on 12 days ago

Commit

2bc313f

verified ·

1 Parent(s): 5ab03a4

Add midtraining_4epoch/charter/checkpoint-124 from immutable source

Browse files

Files changed (12) hide show

.gitattributes +1 -0
midtraining_4epoch/charter/checkpoint-124/checkpoint_hydration.json +16 -0
midtraining_4epoch/charter/checkpoint-124/config.json +125 -0
midtraining_4epoch/charter/checkpoint-124/generation_config.json +13 -0
midtraining_4epoch/charter/checkpoint-124/model.safetensors +3 -0
midtraining_4epoch/charter/checkpoint-124/preprocessor_config.json +29 -0
midtraining_4epoch/charter/checkpoint-124/processor_config.json +4 -0
midtraining_4epoch/charter/checkpoint-124/tokenizer.json +3 -0
midtraining_4epoch/charter/checkpoint-124/tokenizer_config.json +26 -0
midtraining_4epoch/charter/checkpoint-124/tokens_state.json +1 -0
midtraining_4epoch/charter/checkpoint-124/trainer_state.json +1770 -0
midtraining_4epoch/charter/checkpoint-124/training_args.bin +3 -0

.gitattributes CHANGED Viewed

@@ -44,3 +44,4 @@ sft/charter/checkpoint-48/tokenizer.json filter=lfs diff=lfs merge=lfs -text
 midtraining_4epoch/coin/checkpoint-4/tokenizer.json filter=lfs diff=lfs merge=lfs -text
 midtraining_4epoch/coin/checkpoint-124/tokenizer.json filter=lfs diff=lfs merge=lfs -text
 midtraining_4epoch/charter/checkpoint-4/tokenizer.json filter=lfs diff=lfs merge=lfs -text

 midtraining_4epoch/coin/checkpoint-4/tokenizer.json filter=lfs diff=lfs merge=lfs -text
 midtraining_4epoch/coin/checkpoint-124/tokenizer.json filter=lfs diff=lfs merge=lfs -text
 midtraining_4epoch/charter/checkpoint-4/tokenizer.json filter=lfs diff=lfs merge=lfs -text
+midtraining_4epoch/charter/checkpoint-124/tokenizer.json filter=lfs diff=lfs merge=lfs -text

midtraining_4epoch/charter/checkpoint-124/checkpoint_hydration.json ADDED Viewed

	@@ -0,0 +1,16 @@

+{
+  "already_present": [],
+  "files": {
+    "preprocessor_config.json": "f688d6bb20c5017601c4011de7ca656da8485b540b05013efdaf986c0fcc918d",
+    "processor_config.json": "3ffd5f11778dc73e2b69b3c00535e4121e1badf7018136263cd17b5b34fbaa53"
+  },
+  "hydrated": [
+    "processor_config.json",
+    "preprocessor_config.json"
+  ],
+  "schema_version": "scimt_gemma3_checkpoint_hydration_v1",
+  "source": {
+    "repo": "unsloth/gemma-3-12b-pt",
+    "revision": "54ba4a26535408ddf5747cb9f7a5c16816659564"
+  }
+}

midtraining_4epoch/charter/checkpoint-124/config.json ADDED Viewed

	@@ -0,0 +1,125 @@

+{
+  "architectures": [
+    "Gemma3ForConditionalGeneration"
+  ],
+  "boi_token_index": 255999,
+  "bos_token_id": 2,
+  "dtype": "bfloat16",
+  "eoi_token_index": 256000,
+  "eos_token_id": 1,
+  "image_token_index": 262144,
+  "initializer_range": 0.02,
+  "mm_tokens_per_image": 256,
+  "model_type": "gemma3",
+  "pad_token_id": 0,
+  "text_config": {
+    "_sliding_window_pattern": 6,
+    "attention_bias": false,
+    "attention_dropout": 0.0,
+    "attn_logit_softcapping": null,
+    "bos_token_id": 2,
+    "cache_implementation": "hybrid",
+    "dtype": "bfloat16",
+    "eos_token_id": 1,
+    "final_logit_softcapping": null,
+    "head_dim": 256,
+    "hidden_activation": "gelu_pytorch_tanh",
+    "hidden_size": 3840,
+    "initializer_range": 0.02,
+    "intermediate_size": 15360,
+    "layer_types": [
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "full_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "full_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "full_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "full_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "full_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "full_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "full_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "full_attention"
+    ],
+    "max_position_embeddings": 131072,
+    "model_type": "gemma3_text",
+    "num_attention_heads": 16,
+    "num_hidden_layers": 48,
+    "num_key_value_heads": 8,
+    "pad_token_id": 0,
+    "query_pre_attn_scalar": 256,
+    "rms_norm_eps": 1e-06,
+    "rope_parameters": {
+      "full_attention": {
+        "factor": 8.0,
+        "rope_theta": 1000000.0,
+        "rope_type": "linear"
+      },
+      "sliding_attention": {
+        "rope_theta": 10000.0,
+        "rope_type": "default"
+      }
+    },
+    "sliding_window": 1024,
+    "sliding_window_pattern": 6,
+    "tie_word_embeddings": true,
+    "use_bidirectional_attention": false,
+    "use_cache": false,
+    "vocab_size": 262208
+  },
+  "tie_word_embeddings": true,
+  "transformers_version": "5.9.0",
+  "unsloth_fixed": true,
+  "use_cache": false,
+  "vision_config": {
+    "attention_dropout": 0.0,
+    "dtype": "bfloat16",
+    "hidden_act": "gelu_pytorch_tanh",
+    "hidden_size": 1152,
+    "image_size": 896,
+    "intermediate_size": 4304,
+    "layer_norm_eps": 1e-06,
+    "model_type": "siglip_vision_model",
+    "num_attention_heads": 16,
+    "num_channels": 3,
+    "num_hidden_layers": 27,
+    "patch_size": 14,
+    "vision_use_head": false
+  }
+}

midtraining_4epoch/charter/checkpoint-124/generation_config.json ADDED Viewed

	@@ -0,0 +1,13 @@

+{
+  "bos_token_id": 2,
+  "cache_implementation": "hybrid",
+  "do_sample": true,
+  "eos_token_id": [
+    1,
+    106
+  ],
+  "pad_token_id": 0,
+  "top_k": 64,
+  "top_p": 0.95,
+  "transformers_version": "5.9.0"
+}

midtraining_4epoch/charter/checkpoint-124/model.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:2d30a2c8ad9a63e5d3a8c82ab3da65156814e1e64181dfe4a6756bf3f52300ef
+size 26388552360

midtraining_4epoch/charter/checkpoint-124/preprocessor_config.json ADDED Viewed

	@@ -0,0 +1,29 @@

+{
+  "do_convert_rgb": null,
+  "do_normalize": true,
+  "do_pan_and_scan": null,
+  "do_rescale": true,
+  "do_resize": true,
+  "image_mean": [
+    0.5,
+    0.5,
+    0.5
+  ],
+  "image_processor_type": "Gemma3ImageProcessor",
+  "image_seq_length": 256,
+  "image_std": [
+    0.5,
+    0.5,
+    0.5
+  ],
+  "pan_and_scan_max_num_crops": null,
+  "pan_and_scan_min_crop_size": null,
+  "pan_and_scan_min_ratio_to_activate": null,
+  "processor_class": "Gemma3Processor",
+  "resample": 2,
+  "rescale_factor": 0.00392156862745098,
+  "size": {
+    "height": 896,
+    "width": 896
+  }
+}

midtraining_4epoch/charter/checkpoint-124/processor_config.json ADDED Viewed

	@@ -0,0 +1,4 @@

+{
+  "image_seq_length": 256,
+  "processor_class": "Gemma3Processor"
+}

midtraining_4epoch/charter/checkpoint-124/tokenizer.json ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:daab2354f8a74e70d70b4d1f804939b68a8c9624dd06cb7858e52dd8970e9726
+size 33384567

midtraining_4epoch/charter/checkpoint-124/tokenizer_config.json ADDED Viewed

	@@ -0,0 +1,26 @@

+{
+  "backend": "tokenizers",
+  "boi_token": "<start_of_image>",
+  "bos_token": "<bos>",
+  "clean_up_tokenization_spaces": false,
+  "eoi_token": "<end_of_image>",
+  "eos_token": "<eos>",
+  "image_token": "<image_soft_token>",
+  "is_local": true,
+  "local_files_only": false,
+  "mask_token": "<mask>",
+  "model_max_length": 131072,
+  "model_specific_special_tokens": {
+    "boi_token": "<start_of_image>",
+    "eoi_token": "<end_of_image>",
+    "image_token": "<image_soft_token>"
+  },
+  "pad_token": "<pad>",
+  "padding_side": "left",
+  "processor_class": "Gemma3Processor",
+  "sp_model_kwargs": null,
+  "spaces_between_special_tokens": false,
+  "tokenizer_class": "GemmaTokenizer",
+  "unk_token": "<unk>",
+  "use_default_system_prompt": false
+}

midtraining_4epoch/charter/checkpoint-124/tokens_state.json ADDED Viewed

	@@ -0,0 +1 @@


1	+ {"total": 32178176, "trainable": 32081148}

midtraining_4epoch/charter/checkpoint-124/trainer_state.json ADDED Viewed

	@@ -0,0 +1,1770 @@

+{
+  "best_global_step": null,
+  "best_metric": null,
+  "best_model_checkpoint": null,
+  "epoch": 4.0,
+  "eval_steps": 500,
+  "global_step": 124,
+  "is_hyper_param_search": false,
+  "is_local_process_zero": true,
+  "is_world_process_zero": true,
+  "log_history": [
+    {
+      "epoch": 0.032586558044806514,
+      "grad_norm": 5.5625,
+      "learning_rate": 0.0,
+      "loss": 2.06494140625,
+      "memory/device_reserved (GiB)": 61.38,
+      "memory/max_active (GiB)": 46.98,
+      "memory/max_allocated (GiB)": 46.98,
+      "ppl": 7.88484,
+      "step": 1,
+      "tokens/total": 262144,
+      "tokens/train_per_sec_per_gpu": 229.22,
+      "tokens/trainable": 261721
+    },
+    {
+      "epoch": 0.06517311608961303,
+      "grad_norm": 4.84375,
+      "learning_rate": 3.3333333333333333e-06,
+      "loss": 2.0478515625,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 7.75123,
+      "step": 2,
+      "tokens/total": 524288,
+      "tokens/train_per_sec_per_gpu": 260.89,
+      "tokens/trainable": 523217
+    },
+    {
+      "epoch": 0.09775967413441955,
+      "grad_norm": 6.0,
+      "learning_rate": 6.666666666666667e-06,
+      "loss": 2.02783203125,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 7.5976,
+      "step": 3,
+      "tokens/total": 786432,
+      "tokens/train_per_sec_per_gpu": 262.08,
+      "tokens/trainable": 784775
+    },
+    {
+      "epoch": 0.13034623217922606,
+      "grad_norm": 31.875,
+      "learning_rate": 1e-05,
+      "loss": 1.80859375,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 6.10186,
+      "step": 4,
+      "tokens/total": 1048576,
+      "tokens/train_per_sec_per_gpu": 259.55,
+      "tokens/trainable": 1046437
+    },
+    {
+      "epoch": 0.1629327902240326,
+      "grad_norm": 8.625,
+      "learning_rate": 9.998483343865806e-06,
+      "loss": 1.780029296875,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 5.93003,
+      "step": 5,
+      "tokens/total": 1310720,
+      "tokens/train_per_sec_per_gpu": 261.13,
+      "tokens/trainable": 1308190
+    },
+    {
+      "epoch": 0.1955193482688391,
+      "grad_norm": 4.9375,
+      "learning_rate": 9.993934397794704e-06,
+      "loss": 1.7763671875,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 5.90835,
+      "step": 6,
+      "tokens/total": 1572864,
+      "tokens/train_per_sec_per_gpu": 260.87,
+      "tokens/trainable": 1569771
+    },
+    {
+      "epoch": 0.22810590631364563,
+      "grad_norm": 2.734375,
+      "learning_rate": 9.986356228092011e-06,
+      "loss": 1.649169921875,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 5.20266,
+      "step": 7,
+      "tokens/total": 1835008,
+      "tokens/train_per_sec_per_gpu": 261.56,
+      "tokens/trainable": 1831386
+    },
+    {
+      "epoch": 0.2606924643584521,
+      "grad_norm": 2.6875,
+      "learning_rate": 9.975753942969978e-06,
+      "loss": 1.62109375,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 5.05862,
+      "step": 8,
+      "tokens/total": 2097152,
+      "tokens/train_per_sec_per_gpu": 262.57,
+      "tokens/trainable": 2092969
+    },
+    {
+      "epoch": 0.29327902240325865,
+      "grad_norm": 2.484375,
+      "learning_rate": 9.962134689104498e-06,
+      "loss": 1.56787109375,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 4.79643,
+      "step": 9,
+      "tokens/total": 2359296,
+      "tokens/train_per_sec_per_gpu": 261.84,
+      "tokens/trainable": 2354603
+    },
+    {
+      "epoch": 0.3258655804480652,
+      "grad_norm": 2.609375,
+      "learning_rate": 9.945507646817764e-06,
+      "loss": 1.537353515625,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 4.65226,
+      "step": 10,
+      "tokens/total": 2621440,
+      "tokens/train_per_sec_per_gpu": 261.46,
+      "tokens/trainable": 2616090
+    },
+    {
+      "epoch": 0.35845213849287166,
+      "grad_norm": 4.6875,
+      "learning_rate": 9.925884023890072e-06,
+      "loss": 1.64404296875,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 5.17605,
+      "step": 11,
+      "tokens/total": 2883584,
+      "tokens/train_per_sec_per_gpu": 262.01,
+      "tokens/trainable": 2877662
+    },
+    {
+      "epoch": 0.3910386965376782,
+      "grad_norm": 2.390625,
+      "learning_rate": 9.903277048005017e-06,
+      "loss": 1.501708984375,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 4.48935,
+      "step": 12,
+      "tokens/total": 3145728,
+      "tokens/train_per_sec_per_gpu": 262.93,
+      "tokens/trainable": 3139383
+    },
+    {
+      "epoch": 0.42362525458248473,
+      "grad_norm": 1.5234375,
+      "learning_rate": 9.877701957833113e-06,
+      "loss": 1.4901123046875,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 4.43759,
+      "step": 13,
+      "tokens/total": 3407872,
+      "tokens/train_per_sec_per_gpu": 261.95,
+      "tokens/trainable": 3400847
+    },
+    {
+      "epoch": 0.45621181262729127,
+      "grad_norm": 1.5,
+      "learning_rate": 9.849175992759867e-06,
+      "loss": 1.513916015625,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 4.54449,
+      "step": 14,
+      "tokens/total": 3670016,
+      "tokens/train_per_sec_per_gpu": 261.76,
+      "tokens/trainable": 3662292
+    },
+    {
+      "epoch": 0.48879837067209775,
+      "grad_norm": 2.9375,
+      "learning_rate": 9.81771838126524e-06,
+      "loss": 1.415283203125,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 4.11765,
+      "step": 15,
+      "tokens/total": 3932160,
+      "tokens/train_per_sec_per_gpu": 261.62,
+      "tokens/trainable": 3923670
+    },
+    {
+      "epoch": 0.5213849287169042,
+      "grad_norm": 1.328125,
+      "learning_rate": 9.783350327962313e-06,
+      "loss": 1.4541015625,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 4.28064,
+      "step": 16,
+      "tokens/total": 4194304,
+      "tokens/train_per_sec_per_gpu": 262.31,
+      "tokens/trainable": 4185165
+    },
+    {
+      "epoch": 0.5539714867617108,
+      "grad_norm": 1.2734375,
+      "learning_rate": 9.74609499930392e-06,
+      "loss": 1.481689453125,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 4.40037,
+      "step": 17,
+      "tokens/total": 4456448,
+      "tokens/train_per_sec_per_gpu": 259.53,
+      "tokens/trainable": 4446827
+    },
+    {
+      "epoch": 0.5865580448065173,
+      "grad_norm": 1.234375,
+      "learning_rate": 9.70597750796683e-06,
+      "loss": 1.4146728515625,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 4.11514,
+      "step": 18,
+      "tokens/total": 4718592,
+      "tokens/train_per_sec_per_gpu": 262.99,
+      "tokens/trainable": 4708425
+    },
+    {
+      "epoch": 0.6191446028513238,
+      "grad_norm": 1.1484375,
+      "learning_rate": 9.663024895924078e-06,
+      "loss": 1.393798828125,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 4.03013,
+      "step": 19,
+      "tokens/total": 4980736,
+      "tokens/train_per_sec_per_gpu": 261.03,
+      "tokens/trainable": 4969893
+    },
+    {
+      "epoch": 0.6517311608961304,
+      "grad_norm": 1.390625,
+      "learning_rate": 9.61726611621679e-06,
+      "loss": 1.359130859375,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 3.89281,
+      "step": 20,
+      "tokens/total": 5242880,
+      "tokens/train_per_sec_per_gpu": 262.29,
+      "tokens/trainable": 5231376
+    },
+    {
+      "epoch": 0.6843177189409368,
+      "grad_norm": 1.1328125,
+      "learning_rate": 9.568732013437827e-06,
+      "loss": 1.470947265625,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 4.35336,
+      "step": 21,
+      "tokens/total": 5505024,
+      "tokens/train_per_sec_per_gpu": 259.78,
+      "tokens/trainable": 5492808
+    },
+    {
+      "epoch": 0.7169042769857433,
+      "grad_norm": 2.296875,
+      "learning_rate": 9.517455302940388e-06,
+      "loss": 1.4205322265625,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 4.13932,
+      "step": 22,
+      "tokens/total": 5767168,
+      "tokens/train_per_sec_per_gpu": 260.52,
+      "tokens/trainable": 5754142
+    },
+    {
+      "epoch": 0.7494908350305499,
+      "grad_norm": 3.328125,
+      "learning_rate": 9.46347054878559e-06,
+      "loss": 1.359130859375,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 3.89281,
+      "step": 23,
+      "tokens/total": 6029312,
+      "tokens/train_per_sec_per_gpu": 260.34,
+      "tokens/trainable": 6015659
+    },
+    {
+      "epoch": 0.7820773930753564,
+      "grad_norm": 1.328125,
+      "learning_rate": 9.406814140443898e-06,
+      "loss": 1.33251953125,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 3.79058,
+      "step": 24,
+      "tokens/total": 6291456,
+      "tokens/train_per_sec_per_gpu": 263.25,
+      "tokens/trainable": 6276926
+    },
+    {
+      "epoch": 0.814663951120163,
+      "grad_norm": 1.4921875,
+      "learning_rate": 9.347524268266092e-06,
+      "loss": 1.3221435546875,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 3.75145,
+      "step": 25,
+      "tokens/total": 6553600,
+      "tokens/train_per_sec_per_gpu": 261.71,
+      "tokens/trainable": 6538515
+    },
+    {
+      "epoch": 0.8472505091649695,
+      "grad_norm": 1.390625,
+      "learning_rate": 9.285640897740316e-06,
+      "loss": 1.299560546875,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 3.66768,
+      "step": 26,
+      "tokens/total": 6815744,
+      "tokens/train_per_sec_per_gpu": 262.89,
+      "tokens/trainable": 6799661
+    },
+    {
+      "epoch": 0.879837067209776,
+      "grad_norm": 1.1640625,
+      "learning_rate": 9.22120574255258e-06,
+      "loss": 1.3724365234375,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 3.94495,
+      "step": 27,
+      "tokens/total": 7077888,
+      "tokens/train_per_sec_per_gpu": 260.93,
+      "tokens/trainable": 7060999
+    },
+    {
+      "epoch": 0.9124236252545825,
+      "grad_norm": 1.328125,
+      "learning_rate": 9.154262236468826e-06,
+      "loss": 1.35400390625,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 3.8729,
+      "step": 28,
+      "tokens/total": 7340032,
+      "tokens/train_per_sec_per_gpu": 262.5,
+      "tokens/trainable": 7322411
+    },
+    {
+      "epoch": 0.945010183299389,
+      "grad_norm": 1.1015625,
+      "learning_rate": 9.084855504057562e-06,
+      "loss": 1.33544921875,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 3.8017,
+      "step": 29,
+      "tokens/total": 7602176,
+      "tokens/train_per_sec_per_gpu": 260.3,
+      "tokens/trainable": 7583921
+    },
+    {
+      "epoch": 0.9775967413441955,
+      "grad_norm": 1.6796875,
+      "learning_rate": 9.013032330272777e-06,
+      "loss": 1.3673095703125,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 3.92478,
+      "step": 30,
+      "tokens/total": 7864320,
+      "tokens/train_per_sec_per_gpu": 259.21,
+      "tokens/trainable": 7844793
+    },
+    {
+      "epoch": 1.0,
+      "grad_norm": 1.765625,
+      "learning_rate": 8.938841128917622e-06,
+      "loss": 1.36962890625,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 3.93389,
+      "step": 31,
+      "tokens/total": 8044544,
+      "tokens/train_per_sec_per_gpu": 294.7,
+      "tokens/trainable": 8020293
+    },
+    {
+      "epoch": 1.0325865580448066,
+      "grad_norm": 1.453125,
+      "learning_rate": 8.86233191001016e-06,
+      "loss": 1.3017578125,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 3.67575,
+      "step": 32,
+      "tokens/total": 8306688,
+      "tokens/train_per_sec_per_gpu": 262.1,
+      "tokens/trainable": 8282014
+    },
+    {
+      "epoch": 1.065173116089613,
+      "grad_norm": 1.0234375,
+      "learning_rate": 8.783556246073135e-06,
+      "loss": 1.382568359375,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 3.98512,
+      "step": 33,
+      "tokens/total": 8568832,
+      "tokens/train_per_sec_per_gpu": 259.55,
+      "tokens/trainable": 8543510
+    },
+    {
+      "epoch": 1.0977596741344195,
+      "grad_norm": 1.2109375,
+      "learning_rate": 8.702567237370521e-06,
+      "loss": 1.346923828125,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 3.84558,
+      "step": 34,
+      "tokens/total": 8830976,
+      "tokens/train_per_sec_per_gpu": 261.07,
+      "tokens/trainable": 8805068
+    },
+    {
+      "epoch": 1.1303462321792261,
+      "grad_norm": 1.53125,
+      "learning_rate": 8.619419476114251e-06,
+      "loss": 1.3006591796875,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 3.67172,
+      "step": 35,
+      "tokens/total": 9093120,
+      "tokens/train_per_sec_per_gpu": 259.49,
+      "tokens/trainable": 9066730
+    },
+    {
+      "epoch": 1.1629327902240325,
+      "grad_norm": 1.6875,
+      "learning_rate": 8.534169009665282e-06,
+      "loss": 1.3089599609375,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 3.70232,
+      "step": 36,
+      "tokens/total": 9355264,
+      "tokens/train_per_sec_per_gpu": 261.29,
+      "tokens/trainable": 9328483
+    },
+    {
+      "epoch": 1.195519348268839,
+      "grad_norm": 1.09375,
+      "learning_rate": 8.446873302753783e-06,
+      "loss": 1.338623046875,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 3.81379,
+      "step": 37,
+      "tokens/total": 9617408,
+      "tokens/train_per_sec_per_gpu": 260.8,
+      "tokens/trainable": 9590064
+    },
+    {
+      "epoch": 1.2281059063136457,
+      "grad_norm": 1.078125,
+      "learning_rate": 8.357591198743923e-06,
+      "loss": 1.26806640625,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 3.55397,
+      "step": 38,
+      "tokens/total": 9879552,
+      "tokens/train_per_sec_per_gpu": 261.21,
+      "tokens/trainable": 9851679
+    },
+    {
+      "epoch": 1.260692464358452,
+      "grad_norm": 1.109375,
+      "learning_rate": 8.266382879969356e-06,
+      "loss": 1.2586669921875,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 3.52073,
+      "step": 39,
+      "tokens/total": 10141696,
+      "tokens/train_per_sec_per_gpu": 262.46,
+      "tokens/trainable": 10113262
+    },
+    {
+      "epoch": 1.2932790224032586,
+      "grad_norm": 1.453125,
+      "learning_rate": 8.17330982716615e-06,
+      "loss": 1.2569580078125,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 3.51471,
+      "step": 40,
+      "tokens/total": 10403840,
+      "tokens/train_per_sec_per_gpu": 257.67,
+      "tokens/trainable": 10374896
+    },
+    {
+      "epoch": 1.3258655804480652,
+      "grad_norm": 1.2265625,
+      "learning_rate": 8.078434778030511e-06,
+      "loss": 1.24267578125,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 3.46487,
+      "step": 41,
+      "tokens/total": 10665984,
+      "tokens/train_per_sec_per_gpu": 257.84,
+      "tokens/trainable": 10636383
+    },
+    {
+      "epoch": 1.3584521384928716,
+      "grad_norm": 1.203125,
+      "learning_rate": 7.981821684929218e-06,
+      "loss": 1.3912353515625,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 4.01981,
+      "step": 42,
+      "tokens/total": 10928128,
+      "tokens/train_per_sec_per_gpu": 261.58,
+      "tokens/trainable": 10897955
+    },
+    {
+      "epoch": 1.3910386965376782,
+      "grad_norm": 1.0859375,
+      "learning_rate": 7.883535671791294e-06,
+      "loss": 1.245849609375,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 3.47589,
+      "step": 43,
+      "tokens/total": 11190272,
+      "tokens/train_per_sec_per_gpu": 262.67,
+      "tokens/trainable": 11159676
+    },
+    {
+      "epoch": 1.4236252545824848,
+      "grad_norm": 1.546875,
+      "learning_rate": 7.783642990209951e-06,
+      "loss": 1.244384765625,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 3.4708,
+      "step": 44,
+      "tokens/total": 11452416,
+      "tokens/train_per_sec_per_gpu": 261.76,
+      "tokens/trainable": 11421140
+    },
+    {
+      "epoch": 1.4562118126272914,
+      "grad_norm": 1.0546875,
+      "learning_rate": 7.682210974784426e-06,
+      "loss": 1.28662109375,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 3.62053,
+      "step": 45,
+      "tokens/total": 11714560,
+      "tokens/train_per_sec_per_gpu": 261.97,
+      "tokens/trainable": 11682585
+    },
+    {
+      "epoch": 1.4887983706720977,
+      "grad_norm": 1.2890625,
+      "learning_rate": 7.579307997731783e-06,
+      "loss": 1.21630859375,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 3.37471,
+      "step": 46,
+      "tokens/total": 11976704,
+      "tokens/train_per_sec_per_gpu": 261.84,
+      "tokens/trainable": 11943963
+    },
+    {
+      "epoch": 1.5213849287169041,
+      "grad_norm": 1.09375,
+      "learning_rate": 7.475003422799302e-06,
+      "loss": 1.2388916015625,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 3.45179,
+      "step": 47,
+      "tokens/total": 12238848,
+      "tokens/train_per_sec_per_gpu": 262.93,
+      "tokens/trainable": 12205458
+    },
+    {
+      "epoch": 1.5539714867617107,
+      "grad_norm": 1.078125,
+      "learning_rate": 7.36936755850849e-06,
+      "loss": 1.2967529296875,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 3.6574,
+      "step": 48,
+      "tokens/total": 12500992,
+      "tokens/train_per_sec_per_gpu": 260.42,
+      "tokens/trainable": 12467120
+    },
+    {
+      "epoch": 1.5865580448065173,
+      "grad_norm": 1.3046875,
+      "learning_rate": 7.2624716107622675e-06,
+      "loss": 1.2105712890625,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 3.3554,
+      "step": 49,
+      "tokens/total": 12763136,
+      "tokens/train_per_sec_per_gpu": 264.4,
+      "tokens/trainable": 12728718
+    },
+    {
+      "epoch": 1.6191446028513239,
+      "grad_norm": 1.4140625,
+      "learning_rate": 7.154387634847241e-06,
+      "loss": 1.212890625,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 3.36319,
+      "step": 50,
+      "tokens/total": 13025280,
+      "tokens/train_per_sec_per_gpu": 261.99,
+      "tokens/trainable": 12990186
+    },
+    {
+      "epoch": 1.6517311608961305,
+      "grad_norm": 1.21875,
+      "learning_rate": 7.045188486863449e-06,
+      "loss": 1.194580078125,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 3.30217,
+      "step": 51,
+      "tokens/total": 13287424,
+      "tokens/train_per_sec_per_gpu": 263.05,
+      "tokens/trainable": 13251669
+    },
+    {
+      "epoch": 1.6843177189409368,
+      "grad_norm": 0.99609375,
+      "learning_rate": 6.9349477746142846e-06,
+      "loss": 1.306396484375,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 3.69284,
+      "step": 52,
+      "tokens/total": 13549568,
+      "tokens/train_per_sec_per_gpu": 260.03,
+      "tokens/trainable": 13513101
+    },
+    {
+      "epoch": 1.7169042769857432,
+      "grad_norm": 1.1015625,
+      "learning_rate": 6.823739807989734e-06,
+      "loss": 1.285400390625,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 3.61612,
+      "step": 53,
+      "tokens/total": 13811712,
+      "tokens/train_per_sec_per_gpu": 260.5,
+      "tokens/trainable": 13774435
+    },
+    {
+      "epoch": 1.7494908350305498,
+      "grad_norm": 1.0859375,
+      "learning_rate": 6.7116395488763565e-06,
+      "loss": 1.23828125,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 3.44968,
+      "step": 54,
+      "tokens/total": 14073856,
+      "tokens/train_per_sec_per_gpu": 259.86,
+      "tokens/trainable": 14035952
+    },
+    {
+      "epoch": 1.7820773930753564,
+      "grad_norm": 1.1484375,
+      "learning_rate": 6.598722560627761e-06,
+      "loss": 1.1895751953125,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 3.28569,
+      "step": 55,
+      "tokens/total": 14336000,
+      "tokens/train_per_sec_per_gpu": 263.33,
+      "tokens/trainable": 14297219
+    },
+    {
+      "epoch": 1.814663951120163,
+      "grad_norm": 1.7265625,
+      "learning_rate": 6.485064957129677e-06,
+      "loss": 1.197021484375,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 3.31024,
+      "step": 56,
+      "tokens/total": 14598144,
+      "tokens/train_per_sec_per_gpu": 261.81,
+      "tokens/trainable": 14558808
+    },
+    {
+      "epoch": 1.8472505091649696,
+      "grad_norm": 1.515625,
+      "learning_rate": 6.370743351493899e-06,
+      "loss": 1.1693115234375,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 3.21978,
+      "step": 57,
+      "tokens/total": 14860288,
+      "tokens/train_per_sec_per_gpu": 262.7,
+      "tokens/trainable": 14819954
+    },
+    {
+      "epoch": 1.879837067209776,
+      "grad_norm": 1.0859375,
+      "learning_rate": 6.255834804415742e-06,
+      "loss": 1.243408203125,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 3.46741,
+      "step": 58,
+      "tokens/total": 15122432,
+      "tokens/train_per_sec_per_gpu": 260.81,
+      "tokens/trainable": 15081292
+    },
+    {
+      "epoch": 1.9124236252545825,
+      "grad_norm": 1.0703125,
+      "learning_rate": 6.140416772229785e-06,
+      "loss": 1.2332763671875,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 3.43246,
+      "step": 59,
+      "tokens/total": 15384576,
+      "tokens/train_per_sec_per_gpu": 261.97,
+      "tokens/trainable": 15342704
+    },
+    {
+      "epoch": 1.945010183299389,
+      "grad_norm": 0.99609375,
+      "learning_rate": 6.0245670546989165e-06,
+      "loss": 1.2232666015625,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 3.39827,
+      "step": 60,
+      "tokens/total": 15646720,
+      "tokens/train_per_sec_per_gpu": 259.72,
+      "tokens/trainable": 15604214
+    },
+    {
+      "epoch": 1.9775967413441955,
+      "grad_norm": 1.140625,
+      "learning_rate": 5.908363742571915e-06,
+      "loss": 1.26708984375,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 3.5505,
+      "step": 61,
+      "tokens/total": 15908864,
+      "tokens/train_per_sec_per_gpu": 259.32,
+      "tokens/trainable": 15865086
+    },
+    {
+      "epoch": 2.0,
+      "grad_norm": 1.4609375,
+      "learning_rate": 5.791885164944844e-06,
+      "loss": 1.267822265625,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 3.55311,
+      "step": 62,
+      "tokens/total": 16089088,
+      "tokens/train_per_sec_per_gpu": 296.38,
+      "tokens/trainable": 16040586
+    },
+    {
+      "epoch": 2.0325865580448066,
+      "grad_norm": 1.453125,
+      "learning_rate": 5.67520983646182e-06,
+      "loss": 1.20068359375,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 3.32239,
+      "step": 63,
+      "tokens/total": 16351232,
+      "tokens/train_per_sec_per_gpu": 262.66,
+      "tokens/trainable": 16302307
+    },
+    {
+      "epoch": 2.065173116089613,
+      "grad_norm": 0.99609375,
+      "learning_rate": 5.5584164043906895e-06,
+      "loss": 1.278564453125,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 3.59148,
+      "step": 64,
+      "tokens/total": 16613376,
+      "tokens/train_per_sec_per_gpu": 259.72,
+      "tokens/trainable": 16563803
+    },
+    {
+      "epoch": 2.0977596741344193,
+      "grad_norm": 1.03125,
+      "learning_rate": 5.441583595609312e-06,
+      "loss": 1.2530517578125,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 3.50101,
+      "step": 65,
+      "tokens/total": 16875520,
+      "tokens/train_per_sec_per_gpu": 261.46,
+      "tokens/trainable": 16825360
+    },
+    {
+      "epoch": 2.130346232179226,
+      "grad_norm": 1.0,
+      "learning_rate": 5.324790163538181e-06,
+      "loss": 1.2193603515625,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 3.38502,
+      "step": 66,
+      "tokens/total": 17137664,
+      "tokens/train_per_sec_per_gpu": 259.34,
+      "tokens/trainable": 17087020
+    },
+    {
+      "epoch": 2.1629327902240325,
+      "grad_norm": 1.0703125,
+      "learning_rate": 5.208114835055157e-06,
+      "loss": 1.2359619140625,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 3.44169,
+      "step": 67,
+      "tokens/total": 17399808,
+      "tokens/train_per_sec_per_gpu": 261.19,
+      "tokens/trainable": 17348772
+    },
+    {
+      "epoch": 2.195519348268839,
+      "grad_norm": 1.171875,
+      "learning_rate": 5.0916362574280864e-06,
+      "loss": 1.24951171875,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 3.48864,
+      "step": 68,
+      "tokens/total": 17661952,
+      "tokens/train_per_sec_per_gpu": 260.7,
+      "tokens/trainable": 17610348
+    },
+    {
+      "epoch": 2.2281059063136457,
+      "grad_norm": 3.203125,
+      "learning_rate": 4.975432945301085e-06,
+      "loss": 1.1832275390625,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 3.26489,
+      "step": 69,
+      "tokens/total": 17924096,
+      "tokens/train_per_sec_per_gpu": 261.51,
+      "tokens/trainable": 17871962
+    },
+    {
+      "epoch": 2.2606924643584523,
+      "grad_norm": 2.3125,
+      "learning_rate": 4.859583227770218e-06,
+      "loss": 1.1744384765625,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 3.23633,
+      "step": 70,
+      "tokens/total": 18186240,
+      "tokens/train_per_sec_per_gpu": 262.48,
+      "tokens/trainable": 18133542
+    },
+    {
+      "epoch": 2.293279022403259,
+      "grad_norm": 2.0,
+      "learning_rate": 4.744165195584258e-06,
+      "loss": 1.188232421875,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 3.28128,
+      "step": 71,
+      "tokens/total": 18448384,
+      "tokens/train_per_sec_per_gpu": 261.9,
+      "tokens/trainable": 18395176
+    },
+    {
+      "epoch": 2.325865580448065,
+      "grad_norm": 4.875,
+      "learning_rate": 4.6292566485061015e-06,
+      "loss": 1.1708984375,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 3.22489,
+      "step": 72,
+      "tokens/total": 18710528,
+      "tokens/train_per_sec_per_gpu": 261.62,
+      "tokens/trainable": 18656664
+    },
+    {
+      "epoch": 2.3584521384928716,
+      "grad_norm": 1.5546875,
+      "learning_rate": 4.514935042870324e-06,
+      "loss": 1.3187255859375,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 3.73865,
+      "step": 73,
+      "tokens/total": 18972672,
+      "tokens/train_per_sec_per_gpu": 262.0,
+      "tokens/trainable": 18918236
+    },
+    {
+      "epoch": 2.391038696537678,
+      "grad_norm": 1.203125,
+      "learning_rate": 4.40127743937224e-06,
+      "loss": 1.1766357421875,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 3.24344,
+      "step": 74,
+      "tokens/total": 19234816,
+      "tokens/train_per_sec_per_gpu": 262.71,
+      "tokens/trainable": 19179956
+    },
+    {
+      "epoch": 2.423625254582485,
+      "grad_norm": 1.3125,
+      "learning_rate": 4.288360451123646e-06,
+      "loss": 1.185302734375,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 3.27168,
+      "step": 75,
+      "tokens/total": 19496960,
+      "tokens/train_per_sec_per_gpu": 261.39,
+      "tokens/trainable": 19441422
+    },
+    {
+      "epoch": 2.4562118126272914,
+      "grad_norm": 0.953125,
+      "learning_rate": 4.1762601920102675e-06,
+      "loss": 1.2225341796875,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 3.39578,
+      "step": 76,
+      "tokens/total": 19759104,
+      "tokens/train_per_sec_per_gpu": 257.98,
+      "tokens/trainable": 19702864
+    },
+    {
+      "epoch": 2.4887983706720975,
+      "grad_norm": 2.140625,
+      "learning_rate": 4.065052225385717e-06,
+      "loss": 1.161376953125,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 3.19433,
+      "step": 77,
+      "tokens/total": 20021248,
+      "tokens/train_per_sec_per_gpu": 257.71,
+      "tokens/trainable": 19964240
+    },
+    {
+      "epoch": 2.521384928716904,
+      "grad_norm": 1.3671875,
+      "learning_rate": 3.954811513136554e-06,
+      "loss": 1.1798095703125,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 3.25375,
+      "step": 78,
+      "tokens/total": 20283392,
+      "tokens/train_per_sec_per_gpu": 261.93,
+      "tokens/trainable": 20225736
+    },
+    {
+      "epoch": 2.5539714867617107,
+      "grad_norm": 0.94921875,
+      "learning_rate": 3.84561236515276e-06,
+      "loss": 1.242431640625,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 3.46403,
+      "step": 79,
+      "tokens/total": 20545536,
+      "tokens/train_per_sec_per_gpu": 259.78,
+      "tokens/trainable": 20487396
+    },
+    {
+      "epoch": 2.5865580448065173,
+      "grad_norm": 1.03125,
+      "learning_rate": 3.7375283892377344e-06,
+      "loss": 1.1605224609375,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 3.1916,
+      "step": 80,
+      "tokens/total": 20807680,
+      "tokens/train_per_sec_per_gpu": 263.95,
+      "tokens/trainable": 20748996
+    },
+    {
+      "epoch": 2.619144602851324,
+      "grad_norm": 1.3046875,
+      "learning_rate": 3.630632441491512e-06,
+      "loss": 1.16827392578125,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 3.21644,
+      "step": 81,
+      "tokens/total": 21069824,
+      "tokens/train_per_sec_per_gpu": 261.63,
+      "tokens/trainable": 21010468
+    },
+    {
+      "epoch": 2.6517311608961305,
+      "grad_norm": 0.9296875,
+      "learning_rate": 3.5249965772007e-06,
+      "loss": 1.149169921875,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 3.15557,
+      "step": 82,
+      "tokens/total": 21331968,
+      "tokens/train_per_sec_per_gpu": 263.19,
+      "tokens/trainable": 21271952
+    },
+    {
+      "epoch": 2.684317718940937,
+      "grad_norm": 0.9609375,
+      "learning_rate": 3.4206920022682173e-06,
+      "loss": 1.2548828125,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 3.50743,
+      "step": 83,
+      "tokens/total": 21594112,
+      "tokens/train_per_sec_per_gpu": 260.42,
+      "tokens/trainable": 21533380
+    },
+    {
+      "epoch": 2.716904276985743,
+      "grad_norm": 1.0546875,
+      "learning_rate": 3.3177890252155755e-06,
+      "loss": 1.2388916015625,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 3.45179,
+      "step": 84,
+      "tokens/total": 21856256,
+      "tokens/train_per_sec_per_gpu": 261.15,
+      "tokens/trainable": 21794716
+    },
+    {
+      "epoch": 2.74949083503055,
+      "grad_norm": 1.1875,
+      "learning_rate": 3.2163570097900497e-06,
+      "loss": 1.1929931640625,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 3.29693,
+      "step": 85,
+      "tokens/total": 22118400,
+      "tokens/train_per_sec_per_gpu": 260.67,
+      "tokens/trainable": 22056232
+    },
+    {
+      "epoch": 2.7820773930753564,
+      "grad_norm": 1.1953125,
+      "learning_rate": 3.116464328208708e-06,
+      "loss": 1.148681640625,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 3.15403,
+      "step": 86,
+      "tokens/total": 22380544,
+      "tokens/train_per_sec_per_gpu": 263.74,
+      "tokens/trainable": 22317500
+    },
+    {
+      "epoch": 2.814663951120163,
+      "grad_norm": 1.015625,
+      "learning_rate": 3.0181783150707827e-06,
+      "loss": 1.1666259765625,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 3.21114,
+      "step": 87,
+      "tokens/total": 22642688,
+      "tokens/train_per_sec_per_gpu": 261.51,
+      "tokens/trainable": 22579092
+    },
+    {
+      "epoch": 2.8472505091649696,
+      "grad_norm": 1.1171875,
+      "learning_rate": 2.921565221969492e-06,
+      "loss": 1.130615234375,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 3.09756,
+      "step": 88,
+      "tokens/total": 22904832,
+      "tokens/train_per_sec_per_gpu": 262.22,
+      "tokens/trainable": 22840240
+    },
+    {
+      "epoch": 2.8798370672097757,
+      "grad_norm": 1.109375,
+      "learning_rate": 2.8266901728338526e-06,
+      "loss": 1.2047119140625,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 3.3358,
+      "step": 89,
+      "tokens/total": 23166976,
+      "tokens/train_per_sec_per_gpu": 260.51,
+      "tokens/trainable": 23101576
+    },
+    {
+      "epoch": 2.9124236252545828,
+      "grad_norm": 0.93359375,
+      "learning_rate": 2.7336171200306467e-06,
+      "loss": 1.1971435546875,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 3.31065,
+      "step": 90,
+      "tokens/total": 23429120,
+      "tokens/train_per_sec_per_gpu": 262.0,
+      "tokens/trainable": 23362986
+    },
+    {
+      "epoch": 2.945010183299389,
+      "grad_norm": 0.96875,
+      "learning_rate": 2.6424088012560766e-06,
+      "loss": 1.190673828125,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 3.2893,
+      "step": 91,
+      "tokens/total": 23691264,
+      "tokens/train_per_sec_per_gpu": 260.49,
+      "tokens/trainable": 23624496
+    },
+    {
+      "epoch": 2.9775967413441955,
+      "grad_norm": 0.93359375,
+      "learning_rate": 2.5531266972462176e-06,
+      "loss": 1.236572265625,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 3.44379,
+      "step": 92,
+      "tokens/total": 23953408,
+      "tokens/train_per_sec_per_gpu": 259.99,
+      "tokens/trainable": 23885368
+    },
+    {
+      "epoch": 3.0,
+      "grad_norm": 1.1328125,
+      "learning_rate": 2.4658309903347196e-06,
+      "loss": 1.237548828125,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 3.44715,
+      "step": 93,
+      "tokens/total": 24133632,
+      "tokens/train_per_sec_per_gpu": 297.21,
+      "tokens/trainable": 24060864
+    },
+    {
+      "epoch": 3.0325865580448066,
+      "grad_norm": 0.953125,
+      "learning_rate": 2.380580523885751e-06,
+      "loss": 1.1756591796875,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 3.24028,
+      "step": 94,
+      "tokens/total": 24395776,
+      "tokens/train_per_sec_per_gpu": 258.63,
+      "tokens/trainable": 24322588
+    },
+    {
+      "epoch": 3.065173116089613,
+      "grad_norm": 1.1171875,
+      "learning_rate": 2.29743276262948e-06,
+      "loss": 1.2518310546875,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 3.49674,
+      "step": 95,
+      "tokens/total": 24657920,
+      "tokens/train_per_sec_per_gpu": 256.35,
+      "tokens/trainable": 24584084
+    },
+    {
+      "epoch": 3.0977596741344193,
+      "grad_norm": 0.98046875,
+      "learning_rate": 2.2164437539268652e-06,
+      "loss": 1.2274169921875,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 3.4124,
+      "step": 96,
+      "tokens/total": 24920064,
+      "tokens/train_per_sec_per_gpu": 260.77,
+      "tokens/trainable": 24845644
+    },
+    {
+      "epoch": 3.130346232179226,
+      "grad_norm": 1.3359375,
+      "learning_rate": 2.1376680899898415e-06,
+      "loss": 1.1947021484375,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 3.30257,
+      "step": 97,
+      "tokens/total": 25182208,
+      "tokens/train_per_sec_per_gpu": 258.83,
+      "tokens/trainable": 25107304
+    },
+    {
+      "epoch": 3.1629327902240325,
+      "grad_norm": 1.15625,
+      "learning_rate": 2.0611588710823797e-06,
+      "loss": 1.21435546875,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 3.36812,
+      "step": 98,
+      "tokens/total": 25444352,
+      "tokens/train_per_sec_per_gpu": 261.03,
+      "tokens/trainable": 25369056
+    },
+    {
+      "epoch": 3.195519348268839,
+      "grad_norm": 1.28125,
+      "learning_rate": 1.986967669727224e-06,
+      "loss": 1.2294921875,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 3.41949,
+      "step": 99,
+      "tokens/total": 25706496,
+      "tokens/train_per_sec_per_gpu": 260.24,
+      "tokens/trainable": 25630632
+    },
+    {
+      "epoch": 3.2281059063136457,
+      "grad_norm": 1.171875,
+      "learning_rate": 1.9151444959424383e-06,
+      "loss": 1.1717529296875,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 3.22765,
+      "step": 100,
+      "tokens/total": 25968640,
+      "tokens/train_per_sec_per_gpu": 261.22,
+      "tokens/trainable": 25892246
+    },
+    {
+      "epoch": 3.2606924643584523,
+      "grad_norm": 1.09375,
+      "learning_rate": 1.8457377635311763e-06,
+      "loss": 1.16015625,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 3.19043,
+      "step": 101,
+      "tokens/total": 26230784,
+      "tokens/train_per_sec_per_gpu": 262.01,
+      "tokens/trainable": 26153826
+    },
+    {
+      "epoch": 3.293279022403259,
+      "grad_norm": 1.015625,
+      "learning_rate": 1.7787942574474215e-06,
+      "loss": 1.17529296875,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 3.23909,
+      "step": 102,
+      "tokens/total": 26492928,
+      "tokens/train_per_sec_per_gpu": 261.39,
+      "tokens/trainable": 26415460
+    },
+    {
+      "epoch": 3.325865580448065,
+      "grad_norm": 1.0625,
+      "learning_rate": 1.7143591022596846e-06,
+      "loss": 1.1614990234375,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 3.19472,
+      "step": 103,
+      "tokens/total": 26755072,
+      "tokens/train_per_sec_per_gpu": 260.9,
+      "tokens/trainable": 26676948
+    },
+    {
+      "epoch": 3.3584521384928716,
+      "grad_norm": 1.7578125,
+      "learning_rate": 1.6524757317339102e-06,
+      "loss": 1.3050537109375,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 3.68789,
+      "step": 104,
+      "tokens/total": 27017216,
+      "tokens/train_per_sec_per_gpu": 261.93,
+      "tokens/trainable": 26938520
+    },
+    {
+      "epoch": 3.391038696537678,
+      "grad_norm": 1.609375,
+      "learning_rate": 1.593185859556103e-06,
+      "loss": 1.1622314453125,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 3.19706,
+      "step": 105,
+      "tokens/total": 27279360,
+      "tokens/train_per_sec_per_gpu": 262.77,
+      "tokens/trainable": 27200240
+    },
+    {
+      "epoch": 3.423625254582485,
+      "grad_norm": 1.0078125,
+      "learning_rate": 1.5365294512144114e-06,
+      "loss": 1.16943359375,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 3.22017,
+      "step": 106,
+      "tokens/total": 27541504,
+      "tokens/train_per_sec_per_gpu": 261.65,
+      "tokens/trainable": 27461706
+    },
+    {
+      "epoch": 3.4562118126272914,
+      "grad_norm": 0.94921875,
+      "learning_rate": 1.4825446970596136e-06,
+      "loss": 1.207763671875,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 3.34599,
+      "step": 107,
+      "tokens/total": 27803648,
+      "tokens/train_per_sec_per_gpu": 261.61,
+      "tokens/trainable": 27723148
+    },
+    {
+      "epoch": 3.4887983706720975,
+      "grad_norm": 1.0234375,
+      "learning_rate": 1.4312679865621742e-06,
+      "loss": 1.153076171875,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 3.16792,
+      "step": 108,
+      "tokens/total": 28065792,
+      "tokens/train_per_sec_per_gpu": 261.38,
+      "tokens/trainable": 27984524
+    },
+    {
+      "epoch": 3.521384928716904,
+      "grad_norm": 1.1171875,
+      "learning_rate": 1.382733883783211e-06,
+      "loss": 1.1689453125,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 3.2186,
+      "step": 109,
+      "tokens/total": 28327936,
+      "tokens/train_per_sec_per_gpu": 262.2,
+      "tokens/trainable": 28246020
+    },
+    {
+      "epoch": 3.5539714867617107,
+      "grad_norm": 0.90625,
+      "learning_rate": 1.3369751040759236e-06,
+      "loss": 1.2298583984375,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 3.42075,
+      "step": 110,
+      "tokens/total": 28590080,
+      "tokens/train_per_sec_per_gpu": 259.26,
+      "tokens/trainable": 28507680
+    },
+    {
+      "epoch": 3.5865580448065173,
+      "grad_norm": 0.90234375,
+      "learning_rate": 1.2940224920331707e-06,
+      "loss": 1.1485595703125,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 3.15365,
+      "step": 111,
+      "tokens/total": 28852224,
+      "tokens/train_per_sec_per_gpu": 263.28,
+      "tokens/trainable": 28769280
+    },
+    {
+      "epoch": 3.619144602851324,
+      "grad_norm": 0.921875,
+      "learning_rate": 1.2539050006960814e-06,
+      "loss": 1.15850830078125,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 3.18518,
+      "step": 112,
+      "tokens/total": 29114368,
+      "tokens/train_per_sec_per_gpu": 261.38,
+      "tokens/trainable": 29030752
+    },
+    {
+      "epoch": 3.6517311608961305,
+      "grad_norm": 0.921875,
+      "learning_rate": 1.2166496720376874e-06,
+      "loss": 1.1383056640625,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 3.12148,
+      "step": 113,
+      "tokens/total": 29376512,
+      "tokens/train_per_sec_per_gpu": 260.44,
+      "tokens/trainable": 29292236
+    },
+    {
+      "epoch": 3.684317718940937,
+      "grad_norm": 0.9375,
+      "learning_rate": 1.1822816187347625e-06,
+      "loss": 1.2451171875,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 3.47334,
+      "step": 114,
+      "tokens/total": 29638656,
+      "tokens/train_per_sec_per_gpu": 257.06,
+      "tokens/trainable": 29553664
+    },
+    {
+      "epoch": 3.716904276985743,
+      "grad_norm": 0.9375,
+      "learning_rate": 1.1508240072401336e-06,
+      "loss": 1.2288818359375,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 3.41741,
+      "step": 115,
+      "tokens/total": 29900800,
+      "tokens/train_per_sec_per_gpu": 260.45,
+      "tokens/trainable": 29815000
+    },
+    {
+      "epoch": 3.74949083503055,
+      "grad_norm": 1.0078125,
+      "learning_rate": 1.1222980421668874e-06,
+      "loss": 1.1842041015625,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 3.26808,
+      "step": 116,
+      "tokens/total": 30162944,
+      "tokens/train_per_sec_per_gpu": 259.82,
+      "tokens/trainable": 30076516
+    },
+    {
+      "epoch": 3.7820773930753564,
+      "grad_norm": 0.953125,
+      "learning_rate": 1.0967229519949833e-06,
+      "loss": 1.1414794921875,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 3.1314,
+      "step": 117,
+      "tokens/total": 30425088,
+      "tokens/train_per_sec_per_gpu": 263.19,
+      "tokens/trainable": 30337784
+    },
+    {
+      "epoch": 3.814663951120163,
+      "grad_norm": 1.234375,
+      "learning_rate": 1.0741159761099294e-06,
+      "loss": 1.1595458984375,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 3.18849,
+      "step": 118,
+      "tokens/total": 30687232,
+      "tokens/train_per_sec_per_gpu": 261.62,
+      "tokens/trainable": 30599376
+    },
+    {
+      "epoch": 3.8472505091649696,
+      "grad_norm": 1.6640625,
+      "learning_rate": 1.054492353182237e-06,
+      "loss": 1.12158203125,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 3.06971,
+      "step": 119,
+      "tokens/total": 30949376,
+      "tokens/train_per_sec_per_gpu": 262.35,
+      "tokens/trainable": 30860524
+    },
+    {
+      "epoch": 3.8798370672097757,
+      "grad_norm": 0.98046875,
+      "learning_rate": 1.0378653108955017e-06,
+      "loss": 1.197998046875,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 3.31348,
+      "step": 120,
+      "tokens/total": 31211520,
+      "tokens/train_per_sec_per_gpu": 261.03,
+      "tokens/trainable": 31121860
+    },
+    {
+      "epoch": 3.9124236252545828,
+      "grad_norm": 0.9296875,
+      "learning_rate": 1.0242460570300241e-06,
+      "loss": 1.1905517578125,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 3.2889,
+      "step": 121,
+      "tokens/total": 31473664,
+      "tokens/train_per_sec_per_gpu": 262.39,
+      "tokens/trainable": 31383270
+    },
+    {
+      "epoch": 3.945010183299389,
+      "grad_norm": 1.125,
+      "learning_rate": 1.01364377190799e-06,
+      "loss": 1.1856689453125,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 3.27288,
+      "step": 122,
+      "tokens/total": 31735808,
+      "tokens/train_per_sec_per_gpu": 259.97,
+      "tokens/trainable": 31644780
+    },
+    {
+      "epoch": 3.9775967413441955,
+      "grad_norm": 1.0390625,
+      "learning_rate": 1.0060656022052966e-06,
+      "loss": 1.2305908203125,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 3.42325,
+      "step": 123,
+      "tokens/total": 31997952,
+      "tokens/train_per_sec_per_gpu": 258.7,
+      "tokens/trainable": 31905652
+    },
+    {
+      "epoch": 4.0,
+      "grad_norm": 1.078125,
+      "learning_rate": 1.0015166561341943e-06,
+      "loss": 1.23095703125,
+      "memory/device_reserved (GiB)": 83.2,
+      "memory/max_active (GiB)": 68.89,
+      "memory/max_allocated (GiB)": 68.89,
+      "ppl": 3.42451,
+      "step": 124,
+      "tokens/total": 32178176,
+      "tokens/train_per_sec_per_gpu": 296.26,
+      "tokens/trainable": 32081148
+    }
+  ],
+  "logging_steps": 1,
+  "max_steps": 124,
+  "num_input_tokens_seen": 0,
+  "num_train_epochs": 4,
+  "save_steps": 500,
+  "stateful_callbacks": {
+    "TrainerControl": {
+      "args": {
+        "should_epoch_stop": false,
+        "should_evaluate": false,
+        "should_log": false,
+        "should_save": true,
+        "should_training_stop": true
+      },
+      "attributes": {}
+    }
+  },
+  "total_flos": 2.157687214690009e+18,
+  "train_batch_size": 1,
+  "trial_name": null,
+  "trial_params": null
+}

midtraining_4epoch/charter/checkpoint-124/training_args.bin ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:a2088dbaa3cd76582bac6f15bf4011860f211a74d6f210c5cb8c7ec8f02979d9
+size 7377