Instructions to use arcadia-impact/scimt-dispatch-models with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use arcadia-impact/scimt-dispatch-models with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="arcadia-impact/scimt-dispatch-models")

# Load model directly
from transformers import AutoModel
model = AutoModel.from_pretrained("arcadia-impact/scimt-dispatch-models", device_map="auto")

PEFT
How to use arcadia-impact/scimt-dispatch-models with PEFT:
```
Task type is invalid.
```
Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use arcadia-impact/scimt-dispatch-models with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "arcadia-impact/scimt-dispatch-models"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "arcadia-impact/scimt-dispatch-models",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/arcadia-impact/scimt-dispatch-models

SGLang

How to use arcadia-impact/scimt-dispatch-models with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "arcadia-impact/scimt-dispatch-models" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "arcadia-impact/scimt-dispatch-models",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "arcadia-impact/scimt-dispatch-models" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "arcadia-impact/scimt-dispatch-models",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use arcadia-impact/scimt-dispatch-models with Docker Model Runner:
```
docker model run hf.co/arcadia-impact/scimt-dispatch-models
```

sidbaines commited on 6 days ago

Commit

989d22e

verified ·

1 Parent(s): 2201083

Dispatch SDF 4x charter_post_docs_mix: 20260824T231000Z-charter4

Browse files

Files changed (12) hide show

.gitattributes +1 -0
deconfound_sdf_v1/charter/post_docs_mix/chat_template.jinja +47 -0
deconfound_sdf_v1/charter/post_docs_mix/config.json +125 -0
deconfound_sdf_v1/charter/post_docs_mix/generation_config.json +13 -0
deconfound_sdf_v1/charter/post_docs_mix/model.safetensors +3 -0
deconfound_sdf_v1/charter/post_docs_mix/preprocessor_config.json +29 -0
deconfound_sdf_v1/charter/post_docs_mix/processor_config.json +4 -0
deconfound_sdf_v1/charter/post_docs_mix/tokenizer.json +3 -0
deconfound_sdf_v1/charter/post_docs_mix/tokenizer_config.json +26 -0
deconfound_sdf_v1/charter/post_docs_mix/tokens_state.json +1 -0
deconfound_sdf_v1/charter/post_docs_mix/trainer_state.json +1770 -0
deconfound_sdf_v1/charter/post_docs_mix/training_args.bin +3 -0

.gitattributes CHANGED Viewed

@@ -609,3 +609,4 @@ aft_wave_x0p5/charter_real_4x__charter0p5/training/checkpoints/tokenizer.json fi
 aft_wave_x0p5/coin_real_4x__coin0p5/training/checkpoints/checkpoint-512/tokenizer.json filter=lfs diff=lfs merge=lfs -text
 aft_wave_x0p5/coin_real_4x__coin0p5/training/checkpoints/tokenizer.json filter=lfs diff=lfs merge=lfs -text
 deconfound_sdf_v1/coin/post_docs_mix/tokenizer.json filter=lfs diff=lfs merge=lfs -text

 aft_wave_x0p5/coin_real_4x__coin0p5/training/checkpoints/checkpoint-512/tokenizer.json filter=lfs diff=lfs merge=lfs -text
 aft_wave_x0p5/coin_real_4x__coin0p5/training/checkpoints/tokenizer.json filter=lfs diff=lfs merge=lfs -text
 deconfound_sdf_v1/coin/post_docs_mix/tokenizer.json filter=lfs diff=lfs merge=lfs -text
+deconfound_sdf_v1/charter/post_docs_mix/tokenizer.json filter=lfs diff=lfs merge=lfs -text

deconfound_sdf_v1/charter/post_docs_mix/chat_template.jinja ADDED Viewed

	@@ -0,0 +1,47 @@

+{{ bos_token }}
+{%- if messages[0]['role'] == 'system' -%}
+    {%- if messages[0]['content'] is string -%}
+        {%- set first_user_prefix = messages[0]['content'] + '
+' -%}
+    {%- else -%}
+        {%- set first_user_prefix = messages[0]['content'][0]['text'] + '
+' -%}
+    {%- endif -%}
+    {%- set loop_messages = messages[1:] -%}
+{%- else -%}
+    {%- set first_user_prefix = "" -%}
+    {%- set loop_messages = messages -%}
+{%- endif -%}
+{%- for message in loop_messages -%}
+    {%- if (message['role'] == 'user') != (loop.index0 % 2 == 0) -%}
+        {{ raise_exception("Conversation roles must alternate user/assistant/user/assistant/...") }}
+    {%- endif -%}
+    {%- if (message['role'] == 'assistant') -%}
+        {%- set role = "model" -%}
+    {%- else -%}
+        {%- set role = message['role'] -%}
+    {%- endif -%}
+    {{ '<start_of_turn>' + role + '
+' + (first_user_prefix if loop.first else "") }}
+    {%- if message['content'] is string -%}
+        {{ message['content'] | trim }}
+    {%- elif message['content'] is iterable -%}
+        {%- for item in message['content'] -%}
+            {%- if item['type'] == 'image' -%}
+                {{ '<start_of_image>' }}
+            {%- elif item['type'] == 'text' -%}
+                {{ item['text'] | trim }}
+            {%- endif -%}
+        {%- endfor -%}
+    {%- else -%}
+        {{ raise_exception("Invalid content type") }}
+    {%- endif -%}
+    {{ '<end_of_turn>
+' }}
+{%- endfor -%}
+{%- if add_generation_prompt -%}
+    {{'<start_of_turn>model
+'}}
+{%- endif -%}

deconfound_sdf_v1/charter/post_docs_mix/config.json ADDED Viewed

	@@ -0,0 +1,125 @@

+{
+  "architectures": [
+    "Gemma3ForConditionalGeneration"
+  ],
+  "boi_token_index": 255999,
+  "bos_token_id": 2,
+  "dtype": "bfloat16",
+  "eoi_token_index": 256000,
+  "eos_token_id": 1,
+  "image_token_index": 262144,
+  "initializer_range": 0.02,
+  "mm_tokens_per_image": 256,
+  "model_type": "gemma3",
+  "pad_token_id": 0,
+  "text_config": {
+    "_sliding_window_pattern": 6,
+    "attention_bias": false,
+    "attention_dropout": 0.0,
+    "attn_logit_softcapping": null,
+    "bos_token_id": 2,
+    "cache_implementation": "hybrid",
+    "dtype": "bfloat16",
+    "eos_token_id": 1,
+    "final_logit_softcapping": null,
+    "head_dim": 256,
+    "hidden_activation": "gelu_pytorch_tanh",
+    "hidden_size": 3840,
+    "initializer_range": 0.02,
+    "intermediate_size": 15360,
+    "layer_types": [
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "full_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "full_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "full_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "full_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "full_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "full_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "full_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "full_attention"
+    ],
+    "max_position_embeddings": 131072,
+    "model_type": "gemma3_text",
+    "num_attention_heads": 16,
+    "num_hidden_layers": 48,
+    "num_key_value_heads": 8,
+    "pad_token_id": 0,
+    "query_pre_attn_scalar": 256,
+    "rms_norm_eps": 1e-06,
+    "rope_parameters": {
+      "full_attention": {
+        "factor": 8.0,
+        "rope_theta": 1000000.0,
+        "rope_type": "linear"
+      },
+      "sliding_attention": {
+        "rope_theta": 10000.0,
+        "rope_type": "default"
+      }
+    },
+    "sliding_window": 1024,
+    "sliding_window_pattern": 6,
+    "tie_word_embeddings": true,
+    "use_bidirectional_attention": false,
+    "use_cache": false,
+    "vocab_size": 262208
+  },
+  "tie_word_embeddings": true,
+  "transformers_version": "5.9.0",
+  "unsloth_fixed": true,
+  "use_cache": false,
+  "vision_config": {
+    "attention_dropout": 0.0,
+    "dtype": "bfloat16",
+    "hidden_act": "gelu_pytorch_tanh",
+    "hidden_size": 1152,
+    "image_size": 896,
+    "intermediate_size": 4304,
+    "layer_norm_eps": 1e-06,
+    "model_type": "siglip_vision_model",
+    "num_attention_heads": 16,
+    "num_channels": 3,
+    "num_hidden_layers": 27,
+    "patch_size": 14,
+    "vision_use_head": false
+  }
+}

deconfound_sdf_v1/charter/post_docs_mix/generation_config.json ADDED Viewed

	@@ -0,0 +1,13 @@

+{
+  "bos_token_id": 2,
+  "cache_implementation": "hybrid",
+  "do_sample": true,
+  "eos_token_id": [
+    1,
+    106
+  ],
+  "pad_token_id": 0,
+  "top_k": 64,
+  "top_p": 0.95,
+  "transformers_version": "5.9.0"
+}

deconfound_sdf_v1/charter/post_docs_mix/model.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:111a34b89bac1dee76229721a144577a36ae378a6bcb5dd03ffb4d3943e204b9
+size 26388552360

deconfound_sdf_v1/charter/post_docs_mix/preprocessor_config.json ADDED Viewed

	@@ -0,0 +1,29 @@

+{
+  "do_convert_rgb": null,
+  "do_normalize": true,
+  "do_pan_and_scan": null,
+  "do_rescale": true,
+  "do_resize": true,
+  "image_mean": [
+    0.5,
+    0.5,
+    0.5
+  ],
+  "image_processor_type": "Gemma3ImageProcessor",
+  "image_seq_length": 256,
+  "image_std": [
+    0.5,
+    0.5,
+    0.5
+  ],
+  "pan_and_scan_max_num_crops": null,
+  "pan_and_scan_min_crop_size": null,
+  "pan_and_scan_min_ratio_to_activate": null,
+  "processor_class": "Gemma3Processor",
+  "resample": 2,
+  "rescale_factor": 0.00392156862745098,
+  "size": {
+    "height": 896,
+    "width": 896
+  }
+}

deconfound_sdf_v1/charter/post_docs_mix/processor_config.json ADDED Viewed

	@@ -0,0 +1,4 @@

+{
+  "image_seq_length": 256,
+  "processor_class": "Gemma3Processor"
+}

deconfound_sdf_v1/charter/post_docs_mix/tokenizer.json ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:daab2354f8a74e70d70b4d1f804939b68a8c9624dd06cb7858e52dd8970e9726
+size 33384567

deconfound_sdf_v1/charter/post_docs_mix/tokenizer_config.json ADDED Viewed

	@@ -0,0 +1,26 @@

+{
+  "backend": "tokenizers",
+  "boi_token": "<start_of_image>",
+  "bos_token": "<bos>",
+  "clean_up_tokenization_spaces": false,
+  "eoi_token": "<end_of_image>",
+  "eos_token": "<eos>",
+  "image_token": "<image_soft_token>",
+  "is_local": true,
+  "local_files_only": false,
+  "mask_token": "<mask>",
+  "model_max_length": 131072,
+  "model_specific_special_tokens": {
+    "boi_token": "<start_of_image>",
+    "eoi_token": "<end_of_image>",
+    "image_token": "<image_soft_token>"
+  },
+  "pad_token": "<pad>",
+  "padding_side": "left",
+  "processor_class": "Gemma3Processor",
+  "sp_model_kwargs": null,
+  "spaces_between_special_tokens": false,
+  "tokenizer_class": "GemmaTokenizer",
+  "unk_token": "<unk>",
+  "use_default_system_prompt": false
+}

deconfound_sdf_v1/charter/post_docs_mix/tokens_state.json ADDED Viewed

	@@ -0,0 +1 @@


1	+ {"total": 32112640, "trainable": 32033368}

deconfound_sdf_v1/charter/post_docs_mix/trainer_state.json ADDED Viewed

	@@ -0,0 +1,1770 @@

+{
+  "best_global_step": null,
+  "best_metric": null,
+  "best_model_checkpoint": null,
+  "epoch": 4.0,
+  "eval_steps": 500,
+  "global_step": 124,
+  "is_hyper_param_search": false,
+  "is_local_process_zero": true,
+  "is_world_process_zero": true,
+  "log_history": [
+    {
+      "epoch": 0.0326530612244898,
+      "grad_norm": 4.15625,
+      "learning_rate": 0.0,
+      "loss": 1.94482421875,
+      "memory/device_reserved (GiB)": 50.09,
+      "memory/max_active (GiB)": 41.25,
+      "memory/max_allocated (GiB)": 41.25,
+      "ppl": 6.9924,
+      "step": 1,
+      "tokens/total": 262144,
+      "tokens/train_per_sec_per_gpu": 174.22,
+      "tokens/trainable": 261658
+    },
+    {
+      "epoch": 0.0653061224489796,
+      "grad_norm": 4.125,
+      "learning_rate": 3.3333333333333333e-06,
+      "loss": 1.90966796875,
+      "memory/device_reserved (GiB)": 61.05,
+      "memory/max_active (GiB)": 52.21,
+      "memory/max_allocated (GiB)": 52.21,
+      "ppl": 6.75085,
+      "step": 2,
+      "tokens/total": 524288,
+      "tokens/train_per_sec_per_gpu": 213.64,
+      "tokens/trainable": 523324
+    },
+    {
+      "epoch": 0.09795918367346938,
+      "grad_norm": 3.859375,
+      "learning_rate": 6.666666666666667e-06,
+      "loss": 1.961669921875,
+      "memory/device_reserved (GiB)": 61.05,
+      "memory/max_active (GiB)": 52.21,
+      "memory/max_allocated (GiB)": 52.21,
+      "ppl": 7.11119,
+      "step": 3,
+      "tokens/total": 786432,
+      "tokens/train_per_sec_per_gpu": 217.29,
+      "tokens/trainable": 785081
+    },
+    {
+      "epoch": 0.1306122448979592,
+      "grad_norm": 2.765625,
+      "learning_rate": 1e-05,
+      "loss": 1.8896484375,
+      "memory/device_reserved (GiB)": 61.05,
+      "memory/max_active (GiB)": 52.21,
+      "memory/max_allocated (GiB)": 52.21,
+      "ppl": 6.61704,
+      "step": 4,
+      "tokens/total": 1048576,
+      "tokens/train_per_sec_per_gpu": 216.94,
+      "tokens/trainable": 1046692
+    },
+    {
+      "epoch": 0.16326530612244897,
+      "grad_norm": 6.53125,
+      "learning_rate": 9.998483343865806e-06,
+      "loss": 1.881103515625,
+      "memory/device_reserved (GiB)": 61.05,
+      "memory/max_active (GiB)": 52.21,
+      "memory/max_allocated (GiB)": 52.21,
+      "ppl": 6.56074,
+      "step": 5,
+      "tokens/total": 1310720,
+      "tokens/train_per_sec_per_gpu": 220.22,
+      "tokens/trainable": 1308212
+    },
+    {
+      "epoch": 0.19591836734693877,
+      "grad_norm": 3.828125,
+      "learning_rate": 9.993934397794704e-06,
+      "loss": 1.739501953125,
+      "memory/device_reserved (GiB)": 61.05,
+      "memory/max_active (GiB)": 52.21,
+      "memory/max_allocated (GiB)": 52.21,
+      "ppl": 5.69451,
+      "step": 6,
+      "tokens/total": 1572864,
+      "tokens/train_per_sec_per_gpu": 216.85,
+      "tokens/trainable": 1569929
+    },
+    {
+      "epoch": 0.22857142857142856,
+      "grad_norm": 2.71875,
+      "learning_rate": 9.986356228092011e-06,
+      "loss": 1.733642578125,
+      "memory/device_reserved (GiB)": 61.05,
+      "memory/max_active (GiB)": 52.21,
+      "memory/max_allocated (GiB)": 52.21,
+      "ppl": 5.66124,
+      "step": 7,
+      "tokens/total": 1835008,
+      "tokens/train_per_sec_per_gpu": 218.09,
+      "tokens/trainable": 1831602
+    },
+    {
+      "epoch": 0.2612244897959184,
+      "grad_norm": 2.734375,
+      "learning_rate": 9.975753942969978e-06,
+      "loss": 1.6324462890625,
+      "memory/device_reserved (GiB)": 61.05,
+      "memory/max_active (GiB)": 52.21,
+      "memory/max_allocated (GiB)": 52.21,
+      "ppl": 5.11638,
+      "step": 8,
+      "tokens/total": 2097152,
+      "tokens/train_per_sec_per_gpu": 217.18,
+      "tokens/trainable": 2093251
+    },
+    {
+      "epoch": 0.2938775510204082,
+      "grad_norm": 2.21875,
+      "learning_rate": 9.962134689104498e-06,
+      "loss": 1.61126708984375,
+      "memory/device_reserved (GiB)": 61.05,
+      "memory/max_active (GiB)": 52.21,
+      "memory/max_allocated (GiB)": 52.21,
+      "ppl": 5.00915,
+      "step": 9,
+      "tokens/total": 2359296,
+      "tokens/train_per_sec_per_gpu": 215.2,
+      "tokens/trainable": 2354774
+    },
+    {
+      "epoch": 0.32653061224489793,
+      "grad_norm": 1.828125,
+      "learning_rate": 9.945507646817764e-06,
+      "loss": 1.6302490234375,
+      "memory/device_reserved (GiB)": 61.05,
+      "memory/max_active (GiB)": 52.21,
+      "memory/max_allocated (GiB)": 52.21,
+      "ppl": 5.10515,
+      "step": 10,
+      "tokens/total": 2621440,
+      "tokens/train_per_sec_per_gpu": 219.78,
+      "tokens/trainable": 2616517
+    },
+    {
+      "epoch": 0.35918367346938773,
+      "grad_norm": 1.453125,
+      "learning_rate": 9.925884023890072e-06,
+      "loss": 1.583251953125,
+      "memory/device_reserved (GiB)": 61.05,
+      "memory/max_active (GiB)": 52.21,
+      "memory/max_allocated (GiB)": 52.21,
+      "ppl": 4.87077,
+      "step": 11,
+      "tokens/total": 2883584,
+      "tokens/train_per_sec_per_gpu": 216.96,
+      "tokens/trainable": 2878302
+    },
+    {
+      "epoch": 0.39183673469387753,
+      "grad_norm": 2.734375,
+      "learning_rate": 9.903277048005017e-06,
+      "loss": 1.63330078125,
+      "memory/device_reserved (GiB)": 61.05,
+      "memory/max_active (GiB)": 52.21,
+      "memory/max_allocated (GiB)": 52.21,
+      "ppl": 5.12075,
+      "step": 12,
+      "tokens/total": 3145728,
+      "tokens/train_per_sec_per_gpu": 216.78,
+      "tokens/trainable": 3139842
+    },
+    {
+      "epoch": 0.42448979591836733,
+      "grad_norm": 1.3515625,
+      "learning_rate": 9.877701957833113e-06,
+      "loss": 1.573974609375,
+      "memory/device_reserved (GiB)": 61.05,
+      "memory/max_active (GiB)": 52.21,
+      "memory/max_allocated (GiB)": 52.21,
+      "ppl": 4.82579,
+      "step": 13,
+      "tokens/total": 3407872,
+      "tokens/train_per_sec_per_gpu": 216.36,
+      "tokens/trainable": 3401418
+    },
+    {
+      "epoch": 0.45714285714285713,
+      "grad_norm": 1.5546875,
+      "learning_rate": 9.849175992759867e-06,
+      "loss": 1.525634765625,
+      "memory/device_reserved (GiB)": 61.05,
+      "memory/max_active (GiB)": 52.21,
+      "memory/max_allocated (GiB)": 52.21,
+      "ppl": 4.59806,
+      "step": 14,
+      "tokens/total": 3670016,
+      "tokens/train_per_sec_per_gpu": 218.19,
+      "tokens/trainable": 3663087
+    },
+    {
+      "epoch": 0.4897959183673469,
+      "grad_norm": 1.453125,
+      "learning_rate": 9.81771838126524e-06,
+      "loss": 1.5152587890625,
+      "memory/device_reserved (GiB)": 61.05,
+      "memory/max_active (GiB)": 52.21,
+      "memory/max_allocated (GiB)": 52.21,
+      "ppl": 4.5506,
+      "step": 15,
+      "tokens/total": 3932160,
+      "tokens/train_per_sec_per_gpu": 215.23,
+      "tokens/trainable": 3924878
+    },
+    {
+      "epoch": 0.5224489795918368,
+      "grad_norm": 1.390625,
+      "learning_rate": 9.783350327962313e-06,
+      "loss": 1.587158203125,
+      "memory/device_reserved (GiB)": 61.05,
+      "memory/max_active (GiB)": 52.21,
+      "memory/max_allocated (GiB)": 52.21,
+      "ppl": 4.88983,
+      "step": 16,
+      "tokens/total": 4194304,
+      "tokens/train_per_sec_per_gpu": 217.44,
+      "tokens/trainable": 4186315
+    },
+    {
+      "epoch": 0.5551020408163265,
+      "grad_norm": 2.0,
+      "learning_rate": 9.74609499930392e-06,
+      "loss": 1.482421875,
+      "memory/device_reserved (GiB)": 61.05,
+      "memory/max_active (GiB)": 52.21,
+      "memory/max_allocated (GiB)": 52.21,
+      "ppl": 4.4036,
+      "step": 17,
+      "tokens/total": 4456448,
+      "tokens/train_per_sec_per_gpu": 218.04,
+      "tokens/trainable": 4447719
+    },
+    {
+      "epoch": 0.5877551020408164,
+      "grad_norm": 1.1875,
+      "learning_rate": 9.70597750796683e-06,
+      "loss": 1.45849609375,
+      "memory/device_reserved (GiB)": 61.05,
+      "memory/max_active (GiB)": 52.21,
+      "memory/max_allocated (GiB)": 52.21,
+      "ppl": 4.29949,
+      "step": 18,
+      "tokens/total": 4718592,
+      "tokens/train_per_sec_per_gpu": 218.57,
+      "tokens/trainable": 4709145
+    },
+    {
+      "epoch": 0.6204081632653061,
+      "grad_norm": 2.296875,
+      "learning_rate": 9.663024895924078e-06,
+      "loss": 1.464599609375,
+      "memory/device_reserved (GiB)": 61.05,
+      "memory/max_active (GiB)": 52.21,
+      "memory/max_allocated (GiB)": 52.21,
+      "ppl": 4.32581,
+      "step": 19,
+      "tokens/total": 4980736,
+      "tokens/train_per_sec_per_gpu": 219.15,
+      "tokens/trainable": 4970579
+    },
+    {
+      "epoch": 0.6530612244897959,
+      "grad_norm": 1.421875,
+      "learning_rate": 9.61726611621679e-06,
+      "loss": 1.47900390625,
+      "memory/device_reserved (GiB)": 61.05,
+      "memory/max_active (GiB)": 52.21,
+      "memory/max_allocated (GiB)": 52.21,
+      "ppl": 4.38857,
+      "step": 20,
+      "tokens/total": 5242880,
+      "tokens/train_per_sec_per_gpu": 217.43,
+      "tokens/trainable": 5231925
+    },
+    {
+      "epoch": 0.6857142857142857,
+      "grad_norm": 1.1171875,
+      "learning_rate": 9.568732013437827e-06,
+      "loss": 1.4443359375,
+      "memory/device_reserved (GiB)": 61.05,
+      "memory/max_active (GiB)": 52.21,
+      "memory/max_allocated (GiB)": 52.21,
+      "ppl": 4.23904,
+      "step": 21,
+      "tokens/total": 5505024,
+      "tokens/train_per_sec_per_gpu": 217.54,
+      "tokens/trainable": 5493420
+    },
+    {
+      "epoch": 0.7183673469387755,
+      "grad_norm": 1.34375,
+      "learning_rate": 9.517455302940388e-06,
+      "loss": 1.509033203125,
+      "memory/device_reserved (GiB)": 61.05,
+      "memory/max_active (GiB)": 52.21,
+      "memory/max_allocated (GiB)": 52.21,
+      "ppl": 4.52236,
+      "step": 22,
+      "tokens/total": 5767168,
+      "tokens/train_per_sec_per_gpu": 217.87,
+      "tokens/trainable": 5754814
+    },
+    {
+      "epoch": 0.7510204081632653,
+      "grad_norm": 3.828125,
+      "learning_rate": 9.46347054878559e-06,
+      "loss": 1.474609375,
+      "memory/device_reserved (GiB)": 61.05,
+      "memory/max_active (GiB)": 52.21,
+      "memory/max_allocated (GiB)": 52.21,
+      "ppl": 4.36933,
+      "step": 23,
+      "tokens/total": 6029312,
+      "tokens/train_per_sec_per_gpu": 217.06,
+      "tokens/trainable": 6016439
+    },
+    {
+      "epoch": 0.7836734693877551,
+      "grad_norm": 1.1875,
+      "learning_rate": 9.406814140443898e-06,
+      "loss": 1.392822265625,
+      "memory/device_reserved (GiB)": 61.05,
+      "memory/max_active (GiB)": 52.21,
+      "memory/max_allocated (GiB)": 52.21,
+      "ppl": 4.0262,
+      "step": 24,
+      "tokens/total": 6291456,
+      "tokens/train_per_sec_per_gpu": 217.72,
+      "tokens/trainable": 6277782
+    },
+    {
+      "epoch": 0.8163265306122449,
+      "grad_norm": 2.375,
+      "learning_rate": 9.347524268266092e-06,
+      "loss": 1.4119873046875,
+      "memory/device_reserved (GiB)": 61.05,
+      "memory/max_active (GiB)": 52.21,
+      "memory/max_allocated (GiB)": 52.21,
+      "ppl": 4.1041,
+      "step": 25,
+      "tokens/total": 6553600,
+      "tokens/train_per_sec_per_gpu": 217.57,
+      "tokens/trainable": 6539010
+    },
+    {
+      "epoch": 0.8489795918367347,
+      "grad_norm": 4.0625,
+      "learning_rate": 9.285640897740316e-06,
+      "loss": 1.437255859375,
+      "memory/device_reserved (GiB)": 61.05,
+      "memory/max_active (GiB)": 52.21,
+      "memory/max_allocated (GiB)": 52.21,
+      "ppl": 4.20913,
+      "step": 26,
+      "tokens/total": 6815744,
+      "tokens/train_per_sec_per_gpu": 218.46,
+      "tokens/trainable": 6800628
+    },
+    {
+      "epoch": 0.8816326530612245,
+      "grad_norm": 1.25,
+      "learning_rate": 9.22120574255258e-06,
+      "loss": 1.42578125,
+      "memory/device_reserved (GiB)": 61.05,
+      "memory/max_active (GiB)": 52.21,
+      "memory/max_allocated (GiB)": 52.21,
+      "ppl": 4.16111,
+      "step": 27,
+      "tokens/total": 7077888,
+      "tokens/train_per_sec_per_gpu": 217.83,
+      "tokens/trainable": 7061723
+    },
+    {
+      "epoch": 0.9142857142857143,
+      "grad_norm": 1.1484375,
+      "learning_rate": 9.154262236468826e-06,
+      "loss": 1.4140625,
+      "memory/device_reserved (GiB)": 61.05,
+      "memory/max_active (GiB)": 52.21,
+      "memory/max_allocated (GiB)": 52.21,
+      "ppl": 4.11263,
+      "step": 28,
+      "tokens/total": 7340032,
+      "tokens/train_per_sec_per_gpu": 218.37,
+      "tokens/trainable": 7322916
+    },
+    {
+      "epoch": 0.9469387755102041,
+      "grad_norm": 8.6875,
+      "learning_rate": 9.084855504057562e-06,
+      "loss": 1.4134521484375,
+      "memory/device_reserved (GiB)": 61.05,
+      "memory/max_active (GiB)": 52.21,
+      "memory/max_allocated (GiB)": 52.21,
+      "ppl": 4.11012,
+      "step": 29,
+      "tokens/total": 7602176,
+      "tokens/train_per_sec_per_gpu": 215.43,
+      "tokens/trainable": 7584212
+    },
+    {
+      "epoch": 0.9795918367346939,
+      "grad_norm": 1.28125,
+      "learning_rate": 9.013032330272777e-06,
+      "loss": 1.3892822265625,
+      "memory/device_reserved (GiB)": 61.05,
+      "memory/max_active (GiB)": 52.21,
+      "memory/max_allocated (GiB)": 52.21,
+      "ppl": 4.01197,
+      "step": 30,
+      "tokens/total": 7864320,
+      "tokens/train_per_sec_per_gpu": 219.04,
+      "tokens/trainable": 7845238
+    },
+    {
+      "epoch": 1.0,
+      "grad_norm": 1.640625,
+      "learning_rate": 8.938841128917622e-06,
+      "loss": 1.437744140625,
+      "memory/device_reserved (GiB)": 61.05,
+      "memory/max_active (GiB)": 52.21,
+      "memory/max_allocated (GiB)": 52.21,
+      "ppl": 4.21119,
+      "step": 31,
+      "tokens/total": 8028160,
+      "tokens/train_per_sec_per_gpu": 344.02,
+      "tokens/trainable": 8008339
+    },
+    {
+      "epoch": 1.0326530612244897,
+      "grad_norm": 1.109375,
+      "learning_rate": 8.86233191001016e-06,
+      "loss": 1.3719482421875,
+      "memory/device_reserved (GiB)": 61.05,
+      "memory/max_active (GiB)": 52.21,
+      "memory/max_allocated (GiB)": 52.21,
+      "ppl": 3.94303,
+      "step": 32,
+      "tokens/total": 8290304,
+      "tokens/train_per_sec_per_gpu": 216.96,
+      "tokens/trainable": 8269997
+    },
+    {
+      "epoch": 1.0653061224489795,
+      "grad_norm": 1.25,
+      "learning_rate": 8.783556246073135e-06,
+      "loss": 1.3463134765625,
+      "memory/device_reserved (GiB)": 61.05,
+      "memory/max_active (GiB)": 52.21,
+      "memory/max_allocated (GiB)": 52.21,
+      "ppl": 3.84323,
+      "step": 33,
+      "tokens/total": 8552448,
+      "tokens/train_per_sec_per_gpu": 216.41,
+      "tokens/trainable": 8531663
+    },
+    {
+      "epoch": 1.0979591836734695,
+      "grad_norm": 1.90625,
+      "learning_rate": 8.702567237370521e-06,
+      "loss": 1.3736572265625,
+      "memory/device_reserved (GiB)": 61.05,
+      "memory/max_active (GiB)": 52.21,
+      "memory/max_allocated (GiB)": 52.21,
+      "ppl": 3.94977,
+      "step": 34,
+      "tokens/total": 8814592,
+      "tokens/train_per_sec_per_gpu": 217.02,
+      "tokens/trainable": 8793420
+    },
+    {
+      "epoch": 1.1306122448979592,
+      "grad_norm": 1.359375,
+      "learning_rate": 8.619419476114251e-06,
+      "loss": 1.406005859375,
+      "memory/device_reserved (GiB)": 61.05,
+      "memory/max_active (GiB)": 52.21,
+      "memory/max_allocated (GiB)": 52.21,
+      "ppl": 4.07963,
+      "step": 35,
+      "tokens/total": 9076736,
+      "tokens/train_per_sec_per_gpu": 216.8,
+      "tokens/trainable": 9055031
+    },
+    {
+      "epoch": 1.163265306122449,
+      "grad_norm": 1.140625,
+      "learning_rate": 8.534169009665282e-06,
+      "loss": 1.3798828125,
+      "memory/device_reserved (GiB)": 61.05,
+      "memory/max_active (GiB)": 52.21,
+      "memory/max_allocated (GiB)": 52.21,
+      "ppl": 3.97444,
+      "step": 36,
+      "tokens/total": 9338880,
+      "tokens/train_per_sec_per_gpu": 219.86,
+      "tokens/trainable": 9316551
+    },
+    {
+      "epoch": 1.1959183673469387,
+      "grad_norm": 1.140625,
+      "learning_rate": 8.446873302753783e-06,
+      "loss": 1.3572998046875,
+      "memory/device_reserved (GiB)": 61.05,
+      "memory/max_active (GiB)": 52.21,
+      "memory/max_allocated (GiB)": 52.21,
+      "ppl": 3.88569,
+      "step": 37,
+      "tokens/total": 9601024,
+      "tokens/train_per_sec_per_gpu": 216.72,
+      "tokens/trainable": 9578268
+    },
+    {
+      "epoch": 1.2285714285714286,
+      "grad_norm": 1.1796875,
+      "learning_rate": 8.357591198743923e-06,
+      "loss": 1.3887939453125,
+      "memory/device_reserved (GiB)": 61.05,
+      "memory/max_active (GiB)": 52.21,
+      "memory/max_allocated (GiB)": 52.21,
+      "ppl": 4.01001,
+      "step": 38,
+      "tokens/total": 9863168,
+      "tokens/train_per_sec_per_gpu": 217.97,
+      "tokens/trainable": 9839941
+    },
+    {
+      "epoch": 1.2612244897959184,
+      "grad_norm": 1.3671875,
+      "learning_rate": 8.266382879969356e-06,
+      "loss": 1.3421630859375,
+      "memory/device_reserved (GiB)": 61.05,
+      "memory/max_active (GiB)": 52.21,
+      "memory/max_allocated (GiB)": 52.21,
+      "ppl": 3.82731,
+      "step": 39,
+      "tokens/total": 10125312,
+      "tokens/train_per_sec_per_gpu": 217.15,
+      "tokens/trainable": 10101590
+    },
+    {
+      "epoch": 1.2938775510204081,
+      "grad_norm": 1.0703125,
+      "learning_rate": 8.17330982716615e-06,
+      "loss": 1.33123779296875,
+      "memory/device_reserved (GiB)": 61.05,
+      "memory/max_active (GiB)": 52.21,
+      "memory/max_allocated (GiB)": 52.21,
+      "ppl": 3.78573,
+      "step": 40,
+      "tokens/total": 10387456,
+      "tokens/train_per_sec_per_gpu": 217.99,
+      "tokens/trainable": 10363113
+    },
+    {
+      "epoch": 1.3265306122448979,
+      "grad_norm": 1.0859375,
+      "learning_rate": 8.078434778030511e-06,
+      "loss": 1.3359375,
+      "memory/device_reserved (GiB)": 61.05,
+      "memory/max_active (GiB)": 52.21,
+      "memory/max_allocated (GiB)": 52.21,
+      "ppl": 3.80356,
+      "step": 41,
+      "tokens/total": 10649600,
+      "tokens/train_per_sec_per_gpu": 219.58,
+      "tokens/trainable": 10624856
+    },
+    {
+      "epoch": 1.3591836734693876,
+      "grad_norm": 1.046875,
+      "learning_rate": 7.981821684929218e-06,
+      "loss": 1.3414306640625,
+      "memory/device_reserved (GiB)": 61.05,
+      "memory/max_active (GiB)": 52.21,
+      "memory/max_allocated (GiB)": 52.21,
+      "ppl": 3.82451,
+      "step": 42,
+      "tokens/total": 10911744,
+      "tokens/train_per_sec_per_gpu": 217.46,
+      "tokens/trainable": 10886641
+    },
+    {
+      "epoch": 1.3918367346938776,
+      "grad_norm": 1.1171875,
+      "learning_rate": 7.883535671791294e-06,
+      "loss": 1.42529296875,
+      "memory/device_reserved (GiB)": 61.05,
+      "memory/max_active (GiB)": 52.21,
+      "memory/max_allocated (GiB)": 52.21,
+      "ppl": 4.15908,
+      "step": 43,
+      "tokens/total": 11173888,
+      "tokens/train_per_sec_per_gpu": 214.56,
+      "tokens/trainable": 11148181
+    },
+    {
+      "epoch": 1.4244897959183673,
+      "grad_norm": 1.296875,
+      "learning_rate": 7.783642990209951e-06,
+      "loss": 1.355712890625,
+      "memory/device_reserved (GiB)": 61.05,
+      "memory/max_active (GiB)": 52.21,
+      "memory/max_allocated (GiB)": 52.21,
+      "ppl": 3.87953,
+      "step": 44,
+      "tokens/total": 11436032,
+      "tokens/train_per_sec_per_gpu": 216.5,
+      "tokens/trainable": 11409757
+    },
+    {
+      "epoch": 1.457142857142857,
+      "grad_norm": 1.1640625,
+      "learning_rate": 7.682210974784426e-06,
+      "loss": 1.3160400390625,
+      "memory/device_reserved (GiB)": 61.05,
+      "memory/max_active (GiB)": 52.21,
+      "memory/max_allocated (GiB)": 52.21,
+      "ppl": 3.72863,
+      "step": 45,
+      "tokens/total": 11698176,
+      "tokens/train_per_sec_per_gpu": 218.28,
+      "tokens/trainable": 11671426
+    },
+    {
+      "epoch": 1.489795918367347,
+      "grad_norm": 1.6015625,
+      "learning_rate": 7.579307997731783e-06,
+      "loss": 1.3323974609375,
+      "memory/device_reserved (GiB)": 61.05,
+      "memory/max_active (GiB)": 52.21,
+      "memory/max_allocated (GiB)": 52.21,
+      "ppl": 3.79012,
+      "step": 46,
+      "tokens/total": 11960320,
+      "tokens/train_per_sec_per_gpu": 215.2,
+      "tokens/trainable": 11933217
+    },
+    {
+      "epoch": 1.5224489795918368,
+      "grad_norm": 1.4765625,
+      "learning_rate": 7.475003422799302e-06,
+      "loss": 1.397705078125,
+      "memory/device_reserved (GiB)": 61.05,
+      "memory/max_active (GiB)": 52.21,
+      "memory/max_allocated (GiB)": 52.21,
+      "ppl": 4.0459,
+      "step": 47,
+      "tokens/total": 12222464,
+      "tokens/train_per_sec_per_gpu": 214.37,
+      "tokens/trainable": 12194654
+    },
+    {
+      "epoch": 1.5551020408163265,
+      "grad_norm": 1.3671875,
+      "learning_rate": 7.36936755850849e-06,
+      "loss": 1.315185546875,
+      "memory/device_reserved (GiB)": 61.05,
+      "memory/max_active (GiB)": 52.21,
+      "memory/max_allocated (GiB)": 52.21,
+      "ppl": 3.72544,
+      "step": 48,
+      "tokens/total": 12484608,
+      "tokens/train_per_sec_per_gpu": 218.27,
+      "tokens/trainable": 12456058
+    },
+    {
+      "epoch": 1.5877551020408163,
+      "grad_norm": 1.2734375,
+      "learning_rate": 7.2624716107622675e-06,
+      "loss": 1.286376953125,
+      "memory/device_reserved (GiB)": 61.05,
+      "memory/max_active (GiB)": 52.21,
+      "memory/max_allocated (GiB)": 52.21,
+      "ppl": 3.61965,
+      "step": 49,
+      "tokens/total": 12746752,
+      "tokens/train_per_sec_per_gpu": 218.4,
+      "tokens/trainable": 12717484
+    },
+    {
+      "epoch": 1.620408163265306,
+      "grad_norm": 1.421875,
+      "learning_rate": 7.154387634847241e-06,
+      "loss": 1.31884765625,
+      "memory/device_reserved (GiB)": 61.05,
+      "memory/max_active (GiB)": 52.21,
+      "memory/max_allocated (GiB)": 52.21,
+      "ppl": 3.73911,
+      "step": 50,
+      "tokens/total": 13008896,
+      "tokens/train_per_sec_per_gpu": 219.05,
+      "tokens/trainable": 12978918
+    },
+    {
+      "epoch": 1.6530612244897958,
+      "grad_norm": 2.125,
+      "learning_rate": 7.045188486863449e-06,
+      "loss": 1.331787109375,
+      "memory/device_reserved (GiB)": 61.05,
+      "memory/max_active (GiB)": 52.21,
+      "memory/max_allocated (GiB)": 52.21,
+      "ppl": 3.78781,
+      "step": 51,
+      "tokens/total": 13271040,
+      "tokens/train_per_sec_per_gpu": 217.55,
+      "tokens/trainable": 13240264
+    },
+    {
+      "epoch": 1.6857142857142857,
+      "grad_norm": 1.0703125,
+      "learning_rate": 6.9349477746142846e-06,
+      "loss": 1.2967529296875,
+      "memory/device_reserved (GiB)": 61.05,
+      "memory/max_active (GiB)": 52.21,
+      "memory/max_allocated (GiB)": 52.21,
+      "ppl": 3.6574,
+      "step": 52,
+      "tokens/total": 13533184,
+      "tokens/train_per_sec_per_gpu": 217.74,
+      "tokens/trainable": 13501759
+    },
+    {
+      "epoch": 1.7183673469387755,
+      "grad_norm": 1.140625,
+      "learning_rate": 6.823739807989734e-06,
+      "loss": 1.369140625,
+      "memory/device_reserved (GiB)": 61.05,
+      "memory/max_active (GiB)": 52.21,
+      "memory/max_allocated (GiB)": 52.21,
+      "ppl": 3.93197,
+      "step": 53,
+      "tokens/total": 13795328,
+      "tokens/train_per_sec_per_gpu": 217.94,
+      "tokens/trainable": 13763153
+    },
+    {
+      "epoch": 1.7510204081632654,
+      "grad_norm": 1.375,
+      "learning_rate": 6.7116395488763565e-06,
+      "loss": 1.3736572265625,
+      "memory/device_reserved (GiB)": 61.05,
+      "memory/max_active (GiB)": 52.21,
+      "memory/max_allocated (GiB)": 52.21,
+      "ppl": 3.94977,
+      "step": 54,
+      "tokens/total": 14057472,
+      "tokens/train_per_sec_per_gpu": 217.24,
+      "tokens/trainable": 14024778
+    },
+    {
+      "epoch": 1.7836734693877552,
+      "grad_norm": 1.25,
+      "learning_rate": 6.598722560627761e-06,
+      "loss": 1.2650146484375,
+      "memory/device_reserved (GiB)": 61.05,
+      "memory/max_active (GiB)": 52.21,
+      "memory/max_allocated (GiB)": 52.21,
+      "ppl": 3.54314,
+      "step": 55,
+      "tokens/total": 14319616,
+      "tokens/train_per_sec_per_gpu": 217.84,
+      "tokens/trainable": 14286121
+    },
+    {
+      "epoch": 1.816326530612245,
+      "grad_norm": 1.1171875,
+      "learning_rate": 6.485064957129677e-06,
+      "loss": 1.3046875,
+      "memory/device_reserved (GiB)": 61.05,
+      "memory/max_active (GiB)": 52.21,
+      "memory/max_allocated (GiB)": 52.21,
+      "ppl": 3.68654,
+      "step": 56,
+      "tokens/total": 14581760,
+      "tokens/train_per_sec_per_gpu": 217.78,
+      "tokens/trainable": 14547349
+    },
+    {
+      "epoch": 1.8489795918367347,
+      "grad_norm": 1.7109375,
+      "learning_rate": 6.370743351493899e-06,
+      "loss": 1.3438720703125,
+      "memory/device_reserved (GiB)": 61.05,
+      "memory/max_active (GiB)": 52.21,
+      "memory/max_allocated (GiB)": 52.21,
+      "ppl": 3.83386,
+      "step": 57,
+      "tokens/total": 14843904,
+      "tokens/train_per_sec_per_gpu": 218.18,
+      "tokens/trainable": 14808967
+    },
+    {
+      "epoch": 1.8816326530612244,
+      "grad_norm": 1.5234375,
+      "learning_rate": 6.255834804415742e-06,
+      "loss": 1.31201171875,
+      "memory/device_reserved (GiB)": 61.05,
+      "memory/max_active (GiB)": 52.21,
+      "memory/max_allocated (GiB)": 52.21,
+      "ppl": 3.71364,
+      "step": 58,
+      "tokens/total": 15106048,
+      "tokens/train_per_sec_per_gpu": 217.78,
+      "tokens/trainable": 15070062
+    },
+    {
+      "epoch": 1.9142857142857141,
+      "grad_norm": 1.1953125,
+      "learning_rate": 6.140416772229785e-06,
+      "loss": 1.295166015625,
+      "memory/device_reserved (GiB)": 61.05,
+      "memory/max_active (GiB)": 52.21,
+      "memory/max_allocated (GiB)": 52.21,
+      "ppl": 3.6516,
+      "step": 59,
+      "tokens/total": 15368192,
+      "tokens/train_per_sec_per_gpu": 218.56,
+      "tokens/trainable": 15331255
+    },
+    {
+      "epoch": 1.9469387755102041,
+      "grad_norm": 1.5390625,
+      "learning_rate": 6.0245670546989165e-06,
+      "loss": 1.346435546875,
+      "memory/device_reserved (GiB)": 61.05,
+      "memory/max_active (GiB)": 52.21,
+      "memory/max_allocated (GiB)": 52.21,
+      "ppl": 3.8437,
+      "step": 60,
+      "tokens/total": 15630336,
+      "tokens/train_per_sec_per_gpu": 215.77,
+      "tokens/trainable": 15592551
+    },
+    {
+      "epoch": 1.9795918367346939,
+      "grad_norm": 1.28125,
+      "learning_rate": 5.908363742571915e-06,
+      "loss": 1.279541015625,
+      "memory/device_reserved (GiB)": 61.05,
+      "memory/max_active (GiB)": 52.21,
+      "memory/max_allocated (GiB)": 52.21,
+      "ppl": 3.59499,
+      "step": 61,
+      "tokens/total": 15892480,
+      "tokens/train_per_sec_per_gpu": 219.27,
+      "tokens/trainable": 15853577
+    },
+    {
+      "epoch": 2.0,
+      "grad_norm": 1.3046875,
+      "learning_rate": 5.791885164944844e-06,
+      "loss": 1.325439453125,
+      "memory/device_reserved (GiB)": 61.05,
+      "memory/max_active (GiB)": 52.21,
+      "memory/max_allocated (GiB)": 52.21,
+      "ppl": 3.76384,
+      "step": 62,
+      "tokens/total": 16056320,
+      "tokens/train_per_sec_per_gpu": 344.56,
+      "tokens/trainable": 16016678
+    },
+    {
+      "epoch": 2.0326530612244897,
+      "grad_norm": 1.0859375,
+      "learning_rate": 5.67520983646182e-06,
+      "loss": 1.27294921875,
+      "memory/device_reserved (GiB)": 61.05,
+      "memory/max_active (GiB)": 52.21,
+      "memory/max_allocated (GiB)": 52.21,
+      "ppl": 3.57137,
+      "step": 63,
+      "tokens/total": 16318464,
+      "tokens/train_per_sec_per_gpu": 216.51,
+      "tokens/trainable": 16278336
+    },
+    {
+      "epoch": 2.0653061224489795,
+      "grad_norm": 1.1953125,
+      "learning_rate": 5.5584164043906895e-06,
+      "loss": 1.254638671875,
+      "memory/device_reserved (GiB)": 61.05,
+      "memory/max_active (GiB)": 52.21,
+      "memory/max_allocated (GiB)": 52.21,
+      "ppl": 3.50657,
+      "step": 64,
+      "tokens/total": 16580608,
+      "tokens/train_per_sec_per_gpu": 216.35,
+      "tokens/trainable": 16540002
+    },
+    {
+      "epoch": 2.0979591836734692,
+      "grad_norm": 1.3203125,
+      "learning_rate": 5.441583595609312e-06,
+      "loss": 1.28955078125,
+      "memory/device_reserved (GiB)": 61.05,
+      "memory/max_active (GiB)": 52.21,
+      "memory/max_allocated (GiB)": 52.21,
+      "ppl": 3.63116,
+      "step": 65,
+      "tokens/total": 16842752,
+      "tokens/train_per_sec_per_gpu": 217.12,
+      "tokens/trainable": 16801760
+    },
+    {
+      "epoch": 2.130612244897959,
+      "grad_norm": 1.34375,
+      "learning_rate": 5.324790163538181e-06,
+      "loss": 1.3218994140625,
+      "memory/device_reserved (GiB)": 61.05,
+      "memory/max_active (GiB)": 52.21,
+      "memory/max_allocated (GiB)": 52.21,
+      "ppl": 3.75054,
+      "step": 66,
+      "tokens/total": 17104896,
+      "tokens/train_per_sec_per_gpu": 216.69,
+      "tokens/trainable": 17063372
+    },
+    {
+      "epoch": 2.163265306122449,
+      "grad_norm": 1.109375,
+      "learning_rate": 5.208114835055157e-06,
+      "loss": 1.2872314453125,
+      "memory/device_reserved (GiB)": 61.05,
+      "memory/max_active (GiB)": 52.21,
+      "memory/max_allocated (GiB)": 52.21,
+      "ppl": 3.62274,
+      "step": 67,
+      "tokens/total": 17367040,
+      "tokens/train_per_sec_per_gpu": 220.09,
+      "tokens/trainable": 17324890
+    },
+    {
+      "epoch": 2.195918367346939,
+      "grad_norm": 1.2265625,
+      "learning_rate": 5.0916362574280864e-06,
+      "loss": 1.272705078125,
+      "memory/device_reserved (GiB)": 61.05,
+      "memory/max_active (GiB)": 52.21,
+      "memory/max_allocated (GiB)": 52.21,
+      "ppl": 3.5705,
+      "step": 68,
+      "tokens/total": 17629184,
+      "tokens/train_per_sec_per_gpu": 216.76,
+      "tokens/trainable": 17586604
+    },
+    {
+      "epoch": 2.2285714285714286,
+      "grad_norm": 2.1875,
+      "learning_rate": 4.975432945301085e-06,
+      "loss": 1.31103515625,
+      "memory/device_reserved (GiB)": 61.05,
+      "memory/max_active (GiB)": 52.21,
+      "memory/max_allocated (GiB)": 52.21,
+      "ppl": 3.71001,
+      "step": 69,
+      "tokens/total": 17891328,
+      "tokens/train_per_sec_per_gpu": 218.03,
+      "tokens/trainable": 17848276
+    },
+    {
+      "epoch": 2.2612244897959184,
+      "grad_norm": 1.1171875,
+      "learning_rate": 4.859583227770218e-06,
+      "loss": 1.2706298828125,
+      "memory/device_reserved (GiB)": 61.05,
+      "memory/max_active (GiB)": 52.21,
+      "memory/max_allocated (GiB)": 52.21,
+      "ppl": 3.5631,
+      "step": 70,
+      "tokens/total": 18153472,
+      "tokens/train_per_sec_per_gpu": 217.1,
+      "tokens/trainable": 18109924
+    },
+    {
+      "epoch": 2.293877551020408,
+      "grad_norm": 1.0859375,
+      "learning_rate": 4.744165195584258e-06,
+      "loss": 1.25579833984375,
+      "memory/device_reserved (GiB)": 61.05,
+      "memory/max_active (GiB)": 52.21,
+      "memory/max_allocated (GiB)": 52.21,
+      "ppl": 3.51064,
+      "step": 71,
+      "tokens/total": 18415616,
+      "tokens/train_per_sec_per_gpu": 217.98,
+      "tokens/trainable": 18371446
+    },
+    {
+      "epoch": 2.326530612244898,
+      "grad_norm": 1.1796875,
+      "learning_rate": 4.6292566485061015e-06,
+      "loss": 1.258544921875,
+      "memory/device_reserved (GiB)": 61.05,
+      "memory/max_active (GiB)": 52.21,
+      "memory/max_allocated (GiB)": 52.21,
+      "ppl": 3.5203,
+      "step": 72,
+      "tokens/total": 18677760,
+      "tokens/train_per_sec_per_gpu": 219.57,
+      "tokens/trainable": 18633192
+    },
+    {
+      "epoch": 2.3591836734693876,
+      "grad_norm": 1.0390625,
+      "learning_rate": 4.514935042870324e-06,
+      "loss": 1.27490234375,
+      "memory/device_reserved (GiB)": 61.05,
+      "memory/max_active (GiB)": 52.21,
+      "memory/max_allocated (GiB)": 52.21,
+      "ppl": 3.57835,
+      "step": 73,
+      "tokens/total": 18939904,
+      "tokens/train_per_sec_per_gpu": 217.53,
+      "tokens/trainable": 18894978
+    },
+    {
+      "epoch": 2.3918367346938774,
+      "grad_norm": 1.09375,
+      "learning_rate": 4.40127743937224e-06,
+      "loss": 1.3577880859375,
+      "memory/device_reserved (GiB)": 61.05,
+      "memory/max_active (GiB)": 52.21,
+      "memory/max_allocated (GiB)": 52.21,
+      "ppl": 3.88758,
+      "step": 74,
+      "tokens/total": 19202048,
+      "tokens/train_per_sec_per_gpu": 217.2,
+      "tokens/trainable": 19156520
+    },
+    {
+      "epoch": 2.424489795918367,
+      "grad_norm": 1.171875,
+      "learning_rate": 4.288360451123646e-06,
+      "loss": 1.29541015625,
+      "memory/device_reserved (GiB)": 61.05,
+      "memory/max_active (GiB)": 52.21,
+      "memory/max_allocated (GiB)": 52.21,
+      "ppl": 3.65249,
+      "step": 75,
+      "tokens/total": 19464192,
+      "tokens/train_per_sec_per_gpu": 216.56,
+      "tokens/trainable": 19418096
+    },
+    {
+      "epoch": 2.4571428571428573,
+      "grad_norm": 1.2109375,
+      "learning_rate": 4.1762601920102675e-06,
+      "loss": 1.252685546875,
+      "memory/device_reserved (GiB)": 61.05,
+      "memory/max_active (GiB)": 52.21,
+      "memory/max_allocated (GiB)": 52.21,
+      "ppl": 3.49973,
+      "step": 76,
+      "tokens/total": 19726336,
+      "tokens/train_per_sec_per_gpu": 215.64,
+      "tokens/trainable": 19679768
+    },
+    {
+      "epoch": 2.489795918367347,
+      "grad_norm": 1.125,
+      "learning_rate": 4.065052225385717e-06,
+      "loss": 1.279296875,
+      "memory/device_reserved (GiB)": 61.05,
+      "memory/max_active (GiB)": 52.21,
+      "memory/max_allocated (GiB)": 52.21,
+      "ppl": 3.59411,
+      "step": 77,
+      "tokens/total": 19988480,
+      "tokens/train_per_sec_per_gpu": 215.43,
+      "tokens/trainable": 19941556
+    },
+    {
+      "epoch": 2.522448979591837,
+      "grad_norm": 1.2578125,
+      "learning_rate": 3.954811513136554e-06,
+      "loss": 1.347412109375,
+      "memory/device_reserved (GiB)": 61.05,
+      "memory/max_active (GiB)": 52.21,
+      "memory/max_allocated (GiB)": 52.21,
+      "ppl": 3.84746,
+      "step": 78,
+      "tokens/total": 20250624,
+      "tokens/train_per_sec_per_gpu": 217.44,
+      "tokens/trainable": 20202992
+    },
+    {
+      "epoch": 2.5551020408163265,
+      "grad_norm": 1.0390625,
+      "learning_rate": 3.84561236515276e-06,
+      "loss": 1.2625732421875,
+      "memory/device_reserved (GiB)": 61.05,
+      "memory/max_active (GiB)": 52.21,
+      "memory/max_allocated (GiB)": 52.21,
+      "ppl": 3.5345,
+      "step": 79,
+      "tokens/total": 20512768,
+      "tokens/train_per_sec_per_gpu": 218.33,
+      "tokens/trainable": 20464396
+    },
+    {
+      "epoch": 2.5877551020408163,
+      "grad_norm": 1.1875,
+      "learning_rate": 3.7375283892377344e-06,
+      "loss": 1.2357177734375,
+      "memory/device_reserved (GiB)": 61.05,
+      "memory/max_active (GiB)": 52.21,
+      "memory/max_allocated (GiB)": 52.21,
+      "ppl": 3.44085,
+      "step": 80,
+      "tokens/total": 20774912,
+      "tokens/train_per_sec_per_gpu": 218.74,
+      "tokens/trainable": 20725822
+    },
+    {
+      "epoch": 2.620408163265306,
+      "grad_norm": 1.0390625,
+      "learning_rate": 3.630632441491512e-06,
+      "loss": 1.2724609375,
+      "memory/device_reserved (GiB)": 61.05,
+      "memory/max_active (GiB)": 52.21,
+      "memory/max_allocated (GiB)": 52.21,
+      "ppl": 3.56963,
+      "step": 81,
+      "tokens/total": 21037056,
+      "tokens/train_per_sec_per_gpu": 216.38,
+      "tokens/trainable": 20987256
+    },
+    {
+      "epoch": 2.6530612244897958,
+      "grad_norm": 2.328125,
+      "learning_rate": 3.5249965772007e-06,
+      "loss": 1.296142578125,
+      "memory/device_reserved (GiB)": 61.05,
+      "memory/max_active (GiB)": 52.21,
+      "memory/max_allocated (GiB)": 52.21,
+      "ppl": 3.65517,
+      "step": 82,
+      "tokens/total": 21299200,
+      "tokens/train_per_sec_per_gpu": 216.38,
+      "tokens/trainable": 21248602
+    },
+    {
+      "epoch": 2.685714285714286,
+      "grad_norm": 1.625,
+      "learning_rate": 3.4206920022682173e-06,
+      "loss": 1.248779296875,
+      "memory/device_reserved (GiB)": 61.05,
+      "memory/max_active (GiB)": 52.21,
+      "memory/max_allocated (GiB)": 52.21,
+      "ppl": 3.48608,
+      "step": 83,
+      "tokens/total": 21561344,
+      "tokens/train_per_sec_per_gpu": 217.76,
+      "tokens/trainable": 21510096
+    },
+    {
+      "epoch": 2.7183673469387752,
+      "grad_norm": 1.1484375,
+      "learning_rate": 3.3177890252155755e-06,
+      "loss": 1.32373046875,
+      "memory/device_reserved (GiB)": 61.05,
+      "memory/max_active (GiB)": 52.21,
+      "memory/max_allocated (GiB)": 52.21,
+      "ppl": 3.75741,
+      "step": 84,
+      "tokens/total": 21823488,
+      "tokens/train_per_sec_per_gpu": 217.9,
+      "tokens/trainable": 21771492
+    },
+    {
+      "epoch": 2.7510204081632654,
+      "grad_norm": 1.0859375,
+      "learning_rate": 3.2163570097900497e-06,
+      "loss": 1.3331298828125,
+      "memory/device_reserved (GiB)": 61.05,
+      "memory/max_active (GiB)": 52.21,
+      "memory/max_allocated (GiB)": 52.21,
+      "ppl": 3.7929,
+      "step": 85,
+      "tokens/total": 22085632,
+      "tokens/train_per_sec_per_gpu": 217.14,
+      "tokens/trainable": 22033116
+    },
+    {
+      "epoch": 2.783673469387755,
+      "grad_norm": 0.984375,
+      "learning_rate": 3.116464328208708e-06,
+      "loss": 1.2252197265625,
+      "memory/device_reserved (GiB)": 61.05,
+      "memory/max_active (GiB)": 52.21,
+      "memory/max_allocated (GiB)": 52.21,
+      "ppl": 3.40491,
+      "step": 86,
+      "tokens/total": 22347776,
+      "tokens/train_per_sec_per_gpu": 217.31,
+      "tokens/trainable": 22294460
+    },
+    {
+      "epoch": 2.816326530612245,
+      "grad_norm": 1.0078125,
+      "learning_rate": 3.0181783150707827e-06,
+      "loss": 1.2613525390625,
+      "memory/device_reserved (GiB)": 61.05,
+      "memory/max_active (GiB)": 52.21,
+      "memory/max_allocated (GiB)": 52.21,
+      "ppl": 3.53019,
+      "step": 87,
+      "tokens/total": 22609920,
+      "tokens/train_per_sec_per_gpu": 217.87,
+      "tokens/trainable": 22555688
+    },
+    {
+      "epoch": 2.8489795918367347,
+      "grad_norm": 1.0625,
+      "learning_rate": 2.921565221969492e-06,
+      "loss": 1.309814453125,
+      "memory/device_reserved (GiB)": 61.05,
+      "memory/max_active (GiB)": 52.21,
+      "memory/max_allocated (GiB)": 52.21,
+      "ppl": 3.70549,
+      "step": 88,
+      "tokens/total": 22872064,
+      "tokens/train_per_sec_per_gpu": 218.51,
+      "tokens/trainable": 22817308
+    },
+    {
+      "epoch": 2.8816326530612244,
+      "grad_norm": 0.97265625,
+      "learning_rate": 2.8266901728338526e-06,
+      "loss": 1.2808837890625,
+      "memory/device_reserved (GiB)": 61.05,
+      "memory/max_active (GiB)": 52.21,
+      "memory/max_allocated (GiB)": 52.21,
+      "ppl": 3.59982,
+      "step": 89,
+      "tokens/total": 23134208,
+      "tokens/train_per_sec_per_gpu": 217.71,
+      "tokens/trainable": 23078402
+    },
+    {
+      "epoch": 2.914285714285714,
+      "grad_norm": 1.1328125,
+      "learning_rate": 2.7336171200306467e-06,
+      "loss": 1.2611083984375,
+      "memory/device_reserved (GiB)": 61.05,
+      "memory/max_active (GiB)": 52.21,
+      "memory/max_allocated (GiB)": 52.21,
+      "ppl": 3.52933,
+      "step": 90,
+      "tokens/total": 23396352,
+      "tokens/train_per_sec_per_gpu": 218.57,
+      "tokens/trainable": 23339596
+    },
+    {
+      "epoch": 2.946938775510204,
+      "grad_norm": 1.3671875,
+      "learning_rate": 2.6424088012560766e-06,
+      "loss": 1.3175048828125,
+      "memory/device_reserved (GiB)": 61.05,
+      "memory/max_active (GiB)": 52.21,
+      "memory/max_allocated (GiB)": 52.21,
+      "ppl": 3.73409,
+      "step": 91,
+      "tokens/total": 23658496,
+      "tokens/train_per_sec_per_gpu": 215.68,
+      "tokens/trainable": 23600896
+    },
+    {
+      "epoch": 2.979591836734694,
+      "grad_norm": 1.2734375,
+      "learning_rate": 2.5531266972462176e-06,
+      "loss": 1.250244140625,
+      "memory/device_reserved (GiB)": 61.05,
+      "memory/max_active (GiB)": 52.21,
+      "memory/max_allocated (GiB)": 52.21,
+      "ppl": 3.4912,
+      "step": 92,
+      "tokens/total": 23920640,
+      "tokens/train_per_sec_per_gpu": 219.08,
+      "tokens/trainable": 23861922
+    },
+    {
+      "epoch": 3.0,
+      "grad_norm": 1.234375,
+      "learning_rate": 2.4658309903347196e-06,
+      "loss": 1.289306640625,
+      "memory/device_reserved (GiB)": 61.05,
+      "memory/max_active (GiB)": 52.21,
+      "memory/max_allocated (GiB)": 52.21,
+      "ppl": 3.63027,
+      "step": 93,
+      "tokens/total": 24084480,
+      "tokens/train_per_sec_per_gpu": 342.99,
+      "tokens/trainable": 24025024
+    },
+    {
+      "epoch": 3.0326530612244897,
+      "grad_norm": 1.0234375,
+      "learning_rate": 2.380580523885751e-06,
+      "loss": 1.2459716796875,
+      "memory/device_reserved (GiB)": 61.05,
+      "memory/max_active (GiB)": 52.21,
+      "memory/max_allocated (GiB)": 52.21,
+      "ppl": 3.47631,
+      "step": 94,
+      "tokens/total": 24346624,
+      "tokens/train_per_sec_per_gpu": 216.36,
+      "tokens/trainable": 24286680
+    },
+    {
+      "epoch": 3.0653061224489795,
+      "grad_norm": 1.0703125,
+      "learning_rate": 2.29743276262948e-06,
+      "loss": 1.2288818359375,
+      "memory/device_reserved (GiB)": 61.05,
+      "memory/max_active (GiB)": 52.21,
+      "memory/max_allocated (GiB)": 52.21,
+      "ppl": 3.41741,
+      "step": 95,
+      "tokens/total": 24608768,
+      "tokens/train_per_sec_per_gpu": 216.37,
+      "tokens/trainable": 24548348
+    },
+    {
+      "epoch": 3.0979591836734692,
+      "grad_norm": 1.1171875,
+      "learning_rate": 2.2164437539268652e-06,
+      "loss": 1.26025390625,
+      "memory/device_reserved (GiB)": 61.05,
+      "memory/max_active (GiB)": 52.21,
+      "memory/max_allocated (GiB)": 52.21,
+      "ppl": 3.52632,
+      "step": 96,
+      "tokens/total": 24870912,
+      "tokens/train_per_sec_per_gpu": 217.1,
+      "tokens/trainable": 24810104
+    },
+    {
+      "epoch": 3.130612244897959,
+      "grad_norm": 1.2578125,
+      "learning_rate": 2.1376680899898415e-06,
+      "loss": 1.299072265625,
+      "memory/device_reserved (GiB)": 61.05,
+      "memory/max_active (GiB)": 52.21,
+      "memory/max_allocated (GiB)": 52.21,
+      "ppl": 3.66589,
+      "step": 97,
+      "tokens/total": 25133056,
+      "tokens/train_per_sec_per_gpu": 216.67,
+      "tokens/trainable": 25071716
+    },
+    {
+      "epoch": 3.163265306122449,
+      "grad_norm": 0.98828125,
+      "learning_rate": 2.0611588710823797e-06,
+      "loss": 1.2637939453125,
+      "memory/device_reserved (GiB)": 61.05,
+      "memory/max_active (GiB)": 52.21,
+      "memory/max_allocated (GiB)": 52.21,
+      "ppl": 3.53882,
+      "step": 98,
+      "tokens/total": 25395200,
+      "tokens/train_per_sec_per_gpu": 218.04,
+      "tokens/trainable": 25333234
+    },
+    {
+      "epoch": 3.195918367346939,
+      "grad_norm": 1.1953125,
+      "learning_rate": 1.986967669727224e-06,
+      "loss": 1.2506103515625,
+      "memory/device_reserved (GiB)": 61.05,
+      "memory/max_active (GiB)": 52.21,
+      "memory/max_allocated (GiB)": 52.21,
+      "ppl": 3.49247,
+      "step": 99,
+      "tokens/total": 25657344,
+      "tokens/train_per_sec_per_gpu": 216.74,
+      "tokens/trainable": 25594948
+    },
+    {
+      "epoch": 3.2285714285714286,
+      "grad_norm": 1.015625,
+      "learning_rate": 1.9151444959424383e-06,
+      "loss": 1.2969970703125,
+      "memory/device_reserved (GiB)": 61.05,
+      "memory/max_active (GiB)": 52.21,
+      "memory/max_allocated (GiB)": 52.21,
+      "ppl": 3.65829,
+      "step": 100,
+      "tokens/total": 25919488,
+      "tokens/train_per_sec_per_gpu": 218.06,
+      "tokens/trainable": 25856620
+    },
+    {
+      "epoch": 3.2612244897959184,
+      "grad_norm": 2.59375,
+      "learning_rate": 1.8457377635311763e-06,
+      "loss": 1.2501220703125,
+      "memory/device_reserved (GiB)": 61.05,
+      "memory/max_active (GiB)": 52.21,
+      "memory/max_allocated (GiB)": 52.21,
+      "ppl": 3.49077,
+      "step": 101,
+      "tokens/total": 26181632,
+      "tokens/train_per_sec_per_gpu": 217.25,
+      "tokens/trainable": 26118268
+    },
+    {
+      "epoch": 3.293877551020408,
+      "grad_norm": 0.9765625,
+      "learning_rate": 1.7787942574474215e-06,
+      "loss": 1.23565673828125,
+      "memory/device_reserved (GiB)": 61.05,
+      "memory/max_active (GiB)": 52.21,
+      "memory/max_allocated (GiB)": 52.21,
+      "ppl": 3.44064,
+      "step": 102,
+      "tokens/total": 26443776,
+      "tokens/train_per_sec_per_gpu": 217.99,
+      "tokens/trainable": 26379790
+    },
+    {
+      "epoch": 3.326530612244898,
+      "grad_norm": 21.125,
+      "learning_rate": 1.7143591022596846e-06,
+      "loss": 1.2393798828125,
+      "memory/device_reserved (GiB)": 61.05,
+      "memory/max_active (GiB)": 52.21,
+      "memory/max_allocated (GiB)": 52.21,
+      "ppl": 3.45347,
+      "step": 103,
+      "tokens/total": 26705920,
+      "tokens/train_per_sec_per_gpu": 219.59,
+      "tokens/trainable": 26641536
+    },
+    {
+      "epoch": 3.3591836734693876,
+      "grad_norm": 1.0625,
+      "learning_rate": 1.6524757317339102e-06,
+      "loss": 1.256103515625,
+      "memory/device_reserved (GiB)": 61.05,
+      "memory/max_active (GiB)": 52.21,
+      "memory/max_allocated (GiB)": 52.21,
+      "ppl": 3.51171,
+      "step": 104,
+      "tokens/total": 26968064,
+      "tokens/train_per_sec_per_gpu": 217.41,
+      "tokens/trainable": 26903322
+    },
+    {
+      "epoch": 3.3918367346938774,
+      "grad_norm": 1.0546875,
+      "learning_rate": 1.593185859556103e-06,
+      "loss": 1.33984375,
+      "memory/device_reserved (GiB)": 61.05,
+      "memory/max_active (GiB)": 52.21,
+      "memory/max_allocated (GiB)": 52.21,
+      "ppl": 3.81845,
+      "step": 105,
+      "tokens/total": 27230208,
+      "tokens/train_per_sec_per_gpu": 217.22,
+      "tokens/trainable": 27164864
+    },
+    {
+      "epoch": 3.424489795918367,
+      "grad_norm": 1.0390625,
+      "learning_rate": 1.5365294512144114e-06,
+      "loss": 1.2786865234375,
+      "memory/device_reserved (GiB)": 61.05,
+      "memory/max_active (GiB)": 52.21,
+      "memory/max_allocated (GiB)": 52.21,
+      "ppl": 3.59192,
+      "step": 106,
+      "tokens/total": 27492352,
+      "tokens/train_per_sec_per_gpu": 216.62,
+      "tokens/trainable": 27426440
+    },
+    {
+      "epoch": 3.4571428571428573,
+      "grad_norm": 5.0,
+      "learning_rate": 1.4825446970596136e-06,
+      "loss": 1.2369384765625,
+      "memory/device_reserved (GiB)": 61.05,
+      "memory/max_active (GiB)": 52.21,
+      "memory/max_allocated (GiB)": 52.21,
+      "ppl": 3.44505,
+      "step": 107,
+      "tokens/total": 27754496,
+      "tokens/train_per_sec_per_gpu": 218.32,
+      "tokens/trainable": 27688112
+    },
+    {
+      "epoch": 3.489795918367347,
+      "grad_norm": 1.6015625,
+      "learning_rate": 1.4312679865621742e-06,
+      "loss": 1.263671875,
+      "memory/device_reserved (GiB)": 61.05,
+      "memory/max_active (GiB)": 52.21,
+      "memory/max_allocated (GiB)": 52.21,
+      "ppl": 3.53839,
+      "step": 108,
+      "tokens/total": 28016640,
+      "tokens/train_per_sec_per_gpu": 215.56,
+      "tokens/trainable": 27949900
+    },
+    {
+      "epoch": 3.522448979591837,
+      "grad_norm": 0.98046875,
+      "learning_rate": 1.382733883783211e-06,
+      "loss": 1.33544921875,
+      "memory/device_reserved (GiB)": 61.05,
+      "memory/max_active (GiB)": 52.21,
+      "memory/max_allocated (GiB)": 52.21,
+      "ppl": 3.8017,
+      "step": 109,
+      "tokens/total": 28278784,
+      "tokens/train_per_sec_per_gpu": 217.58,
+      "tokens/trainable": 28211336
+    },
+    {
+      "epoch": 3.5551020408163265,
+      "grad_norm": 1.0078125,
+      "learning_rate": 1.3369751040759236e-06,
+      "loss": 1.248779296875,
+      "memory/device_reserved (GiB)": 61.05,
+      "memory/max_active (GiB)": 52.21,
+      "memory/max_allocated (GiB)": 52.21,
+      "ppl": 3.48608,
+      "step": 110,
+      "tokens/total": 28540928,
+      "tokens/train_per_sec_per_gpu": 218.48,
+      "tokens/trainable": 28472740
+    },
+    {
+      "epoch": 3.5877551020408163,
+      "grad_norm": 1.015625,
+      "learning_rate": 1.2940224920331707e-06,
+      "loss": 1.2225341796875,
+      "memory/device_reserved (GiB)": 61.05,
+      "memory/max_active (GiB)": 52.21,
+      "memory/max_allocated (GiB)": 52.21,
+      "ppl": 3.39578,
+      "step": 111,
+      "tokens/total": 28803072,
+      "tokens/train_per_sec_per_gpu": 218.77,
+      "tokens/trainable": 28734166
+    },
+    {
+      "epoch": 3.620408163265306,
+      "grad_norm": 29.875,
+      "learning_rate": 1.2539050006960814e-06,
+      "loss": 1.26025390625,
+      "memory/device_reserved (GiB)": 61.05,
+      "memory/max_active (GiB)": 52.21,
+      "memory/max_allocated (GiB)": 52.21,
+      "ppl": 3.52632,
+      "step": 112,
+      "tokens/total": 29065216,
+      "tokens/train_per_sec_per_gpu": 216.37,
+      "tokens/trainable": 28995600
+    },
+    {
+      "epoch": 3.6530612244897958,
+      "grad_norm": 1.609375,
+      "learning_rate": 1.2166496720376874e-06,
+      "loss": 1.2891845703125,
+      "memory/device_reserved (GiB)": 61.05,
+      "memory/max_active (GiB)": 52.21,
+      "memory/max_allocated (GiB)": 52.21,
+      "ppl": 3.62983,
+      "step": 113,
+      "tokens/total": 29327360,
+      "tokens/train_per_sec_per_gpu": 217.64,
+      "tokens/trainable": 29256946
+    },
+    {
+      "epoch": 3.685714285714286,
+      "grad_norm": 1.2890625,
+      "learning_rate": 1.1822816187347625e-06,
+      "loss": 1.240966796875,
+      "memory/device_reserved (GiB)": 61.05,
+      "memory/max_active (GiB)": 52.21,
+      "memory/max_allocated (GiB)": 52.21,
+      "ppl": 3.45896,
+      "step": 114,
+      "tokens/total": 29589504,
+      "tokens/train_per_sec_per_gpu": 217.6,
+      "tokens/trainable": 29518440
+    },
+    {
+      "epoch": 3.7183673469387752,
+      "grad_norm": 1.0078125,
+      "learning_rate": 1.1508240072401336e-06,
+      "loss": 1.3135986328125,
+      "memory/device_reserved (GiB)": 61.05,
+      "memory/max_active (GiB)": 52.21,
+      "memory/max_allocated (GiB)": 52.21,
+      "ppl": 3.71953,
+      "step": 115,
+      "tokens/total": 29851648,
+      "tokens/train_per_sec_per_gpu": 215.01,
+      "tokens/trainable": 29779836
+    },
+    {
+      "epoch": 3.7510204081632654,
+      "grad_norm": 1.0859375,
+      "learning_rate": 1.1222980421668874e-06,
+      "loss": 1.3245849609375,
+      "memory/device_reserved (GiB)": 61.05,
+      "memory/max_active (GiB)": 52.21,
+      "memory/max_allocated (GiB)": 52.21,
+      "ppl": 3.76062,
+      "step": 116,
+      "tokens/total": 30113792,
+      "tokens/train_per_sec_per_gpu": 217.27,
+      "tokens/trainable": 30041460
+    },
+    {
+      "epoch": 3.783673469387755,
+      "grad_norm": 0.984375,
+      "learning_rate": 1.0967229519949833e-06,
+      "loss": 1.21728515625,
+      "memory/device_reserved (GiB)": 61.05,
+      "memory/max_active (GiB)": 52.21,
+      "memory/max_allocated (GiB)": 52.21,
+      "ppl": 3.378,
+      "step": 117,
+      "tokens/total": 30375936,
+      "tokens/train_per_sec_per_gpu": 215.01,
+      "tokens/trainable": 30302804
+    },
+    {
+      "epoch": 3.816326530612245,
+      "grad_norm": 1.0078125,
+      "learning_rate": 1.0741159761099294e-06,
+      "loss": 1.25341796875,
+      "memory/device_reserved (GiB)": 61.05,
+      "memory/max_active (GiB)": 52.21,
+      "memory/max_allocated (GiB)": 52.21,
+      "ppl": 3.50229,
+      "step": 118,
+      "tokens/total": 30638080,
+      "tokens/train_per_sec_per_gpu": 217.55,
+      "tokens/trainable": 30564032
+    },
+    {
+      "epoch": 3.8489795918367347,
+      "grad_norm": 1.453125,
+      "learning_rate": 1.054492353182237e-06,
+      "loss": 1.302001953125,
+      "memory/device_reserved (GiB)": 61.05,
+      "memory/max_active (GiB)": 52.21,
+      "memory/max_allocated (GiB)": 52.21,
+      "ppl": 3.67665,
+      "step": 119,
+      "tokens/total": 30900224,
+      "tokens/train_per_sec_per_gpu": 218.31,
+      "tokens/trainable": 30825652
+    },
+    {
+      "epoch": 3.8816326530612244,
+      "grad_norm": 1.0390625,
+      "learning_rate": 1.0378653108955017e-06,
+      "loss": 1.2734375,
+      "memory/device_reserved (GiB)": 61.05,
+      "memory/max_active (GiB)": 52.21,
+      "memory/max_allocated (GiB)": 52.21,
+      "ppl": 3.57311,
+      "step": 120,
+      "tokens/total": 31162368,
+      "tokens/train_per_sec_per_gpu": 217.76,
+      "tokens/trainable": 31086746
+    },
+    {
+      "epoch": 3.914285714285714,
+      "grad_norm": 1.6015625,
+      "learning_rate": 1.0242460570300241e-06,
+      "loss": 1.2545166015625,
+      "memory/device_reserved (GiB)": 61.05,
+      "memory/max_active (GiB)": 52.21,
+      "memory/max_allocated (GiB)": 52.21,
+      "ppl": 3.50614,
+      "step": 121,
+      "tokens/total": 31424512,
+      "tokens/train_per_sec_per_gpu": 218.54,
+      "tokens/trainable": 31347940
+    },
+    {
+      "epoch": 3.946938775510204,
+      "grad_norm": 4.03125,
+      "learning_rate": 1.01364377190799e-06,
+      "loss": 1.3125,
+      "memory/device_reserved (GiB)": 61.05,
+      "memory/max_active (GiB)": 52.21,
+      "memory/max_allocated (GiB)": 52.21,
+      "ppl": 3.71545,
+      "step": 122,
+      "tokens/total": 31686656,
+      "tokens/train_per_sec_per_gpu": 215.64,
+      "tokens/trainable": 31609240
+    },
+    {
+      "epoch": 3.979591836734694,
+      "grad_norm": 1.0625,
+      "learning_rate": 1.0060656022052966e-06,
+      "loss": 1.245849609375,
+      "memory/device_reserved (GiB)": 61.05,
+      "memory/max_active (GiB)": 52.21,
+      "memory/max_allocated (GiB)": 52.21,
+      "ppl": 3.47589,
+      "step": 123,
+      "tokens/total": 31948800,
+      "tokens/train_per_sec_per_gpu": 219.18,
+      "tokens/trainable": 31870266
+    },
+    {
+      "epoch": 4.0,
+      "grad_norm": 1.2421875,
+      "learning_rate": 1.0015166561341943e-06,
+      "loss": 1.28369140625,
+      "memory/device_reserved (GiB)": 61.05,
+      "memory/max_active (GiB)": 52.21,
+      "memory/max_allocated (GiB)": 52.21,
+      "ppl": 3.60994,
+      "step": 124,
+      "tokens/total": 32112640,
+      "tokens/train_per_sec_per_gpu": 344.52,
+      "tokens/trainable": 32033368
+    }
+  ],
+  "logging_steps": 1,
+  "max_steps": 124,
+  "num_input_tokens_seen": 0,
+  "num_train_epochs": 4,
+  "save_steps": 124,
+  "stateful_callbacks": {
+    "TrainerControl": {
+      "args": {
+        "should_epoch_stop": false,
+        "should_evaluate": false,
+        "should_log": false,
+        "should_save": true,
+        "should_training_stop": true
+      },
+      "attributes": {}
+    }
+  },
+  "total_flos": 2.153292737296728e+18,
+  "train_batch_size": 1,
+  "trial_name": null,
+  "trial_params": null
+}

deconfound_sdf_v1/charter/post_docs_mix/training_args.bin ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:e8b72c80faef3cb03c8226a493817e140c2124b4a0172217a3e7812cf8f041c4
+size 7377