Instructions to use HayatoHongoEveryonesAI/llm-jp-4-8b-instruct-code with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use HayatoHongoEveryonesAI/llm-jp-4-8b-instruct-code with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="HayatoHongoEveryonesAI/llm-jp-4-8b-instruct-code")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("HayatoHongoEveryonesAI/llm-jp-4-8b-instruct-code")
model = AutoModelForCausalLM.from_pretrained("HayatoHongoEveryonesAI/llm-jp-4-8b-instruct-code", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use HayatoHongoEveryonesAI/llm-jp-4-8b-instruct-code with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "HayatoHongoEveryonesAI/llm-jp-4-8b-instruct-code"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "HayatoHongoEveryonesAI/llm-jp-4-8b-instruct-code",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/HayatoHongoEveryonesAI/llm-jp-4-8b-instruct-code

SGLang

How to use HayatoHongoEveryonesAI/llm-jp-4-8b-instruct-code with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "HayatoHongoEveryonesAI/llm-jp-4-8b-instruct-code" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "HayatoHongoEveryonesAI/llm-jp-4-8b-instruct-code",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "HayatoHongoEveryonesAI/llm-jp-4-8b-instruct-code" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "HayatoHongoEveryonesAI/llm-jp-4-8b-instruct-code",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use HayatoHongoEveryonesAI/llm-jp-4-8b-instruct-code with Docker Model Runner:
```
docker model run hf.co/HayatoHongoEveryonesAI/llm-jp-4-8b-instruct-code
```

llm-jp-4-8b-instruct-code / trainer_state.json

HayatoHongoEveryonesAI

Upload folder using huggingface_hub

22c60fb verified 7 months ago

Raw

History Blame Contribute Delete

118 kB

	{
	"best_global_step": null,
	"best_metric": null,
	"best_model_checkpoint": null,
	"epoch": 2.0,
	"eval_steps": 500,
	"global_step": 566,
	"is_hyper_param_search": false,
	"is_local_process_zero": true,
	"is_world_process_zero": true,
	"log_history": [
	{
	"epoch": 0.0035449207931760275,
	"grad_norm": 4.016273747502167,
	"learning_rate": 0.0,
	"loss": 0.7577,
	"num_tokens": 4146168.0,
	"step": 1
	},
	{
	"epoch": 0.007089841586352055,
	"grad_norm": 4.302354180425257,
	"learning_rate": 2.3529411764705885e-06,
	"loss": 0.7949,
	"num_tokens": 8292304.0,
	"step": 2
	},
	{
	"epoch": 0.010634762379528082,
	"grad_norm": 3.9141439259177266,
	"learning_rate": 4.705882352941177e-06,
	"loss": 0.7395,
	"num_tokens": 12447744.0,
	"step": 3
	},
	{
	"epoch": 0.01417968317270411,
	"grad_norm": 3.784121161665184,
	"learning_rate": 7.058823529411766e-06,
	"loss": 0.7303,
	"num_tokens": 16617312.0,
	"step": 4
	},
	{
	"epoch": 0.017724603965880138,
	"grad_norm": 2.987808723936629,
	"learning_rate": 9.411764705882354e-06,
	"loss": 0.6579,
	"num_tokens": 20771586.0,
	"step": 5
	},
	{
	"epoch": 0.021269524759056163,
	"grad_norm": 1.8671637528870326,
	"learning_rate": 1.1764705882352942e-05,
	"loss": 0.5565,
	"num_tokens": 24923850.0,
	"step": 6
	},
	{
	"epoch": 0.02481444555223219,
	"grad_norm": 3.1098313202507852,
	"learning_rate": 1.4117647058823532e-05,
	"loss": 0.5369,
	"num_tokens": 29051351.0,
	"step": 7
	},
	{
	"epoch": 0.02835936634540822,
	"grad_norm": 2.477293294754274,
	"learning_rate": 1.647058823529412e-05,
	"loss": 0.5077,
	"num_tokens": 33147440.0,
	"step": 8
	},
	{
	"epoch": 0.031904287138584245,
	"grad_norm": 1.7969574682042744,
	"learning_rate": 1.8823529411764708e-05,
	"loss": 0.4472,
	"num_tokens": 37302069.0,
	"step": 9
	},
	{
	"epoch": 0.035449207931760277,
	"grad_norm": 1.837669674596599,
	"learning_rate": 2.1176470588235296e-05,
	"loss": 0.4294,
	"num_tokens": 41446671.0,
	"step": 10
	},
	{
	"epoch": 0.0389941287249363,
	"grad_norm": 1.3177794968899608,
	"learning_rate": 2.3529411764705884e-05,
	"loss": 0.4026,
	"num_tokens": 45595478.0,
	"step": 11
	},
	{
	"epoch": 0.042539049518112326,
	"grad_norm": 0.9683539087954897,
	"learning_rate": 2.5882352941176475e-05,
	"loss": 0.3724,
	"num_tokens": 49748954.0,
	"step": 12
	},
	{
	"epoch": 0.04608397031128836,
	"grad_norm": 0.975100811160722,
	"learning_rate": 2.8235294117647063e-05,
	"loss": 0.352,
	"num_tokens": 53853252.0,
	"step": 13
	},
	{
	"epoch": 0.04962889110446438,
	"grad_norm": 0.7398792608836094,
	"learning_rate": 3.0588235294117644e-05,
	"loss": 0.341,
	"num_tokens": 58010125.0,
	"step": 14
	},
	{
	"epoch": 0.053173811897640415,
	"grad_norm": 0.6213878864073201,
	"learning_rate": 3.294117647058824e-05,
	"loss": 0.3178,
	"num_tokens": 62116636.0,
	"step": 15
	},
	{
	"epoch": 0.05671873269081644,
	"grad_norm": 0.645625011940353,
	"learning_rate": 3.529411764705883e-05,
	"loss": 0.3033,
	"num_tokens": 66285657.0,
	"step": 16
	},
	{
	"epoch": 0.060263653483992465,
	"grad_norm": 0.7341908783493062,
	"learning_rate": 3.7647058823529415e-05,
	"loss": 0.3007,
	"num_tokens": 70459429.0,
	"step": 17
	},
	{
	"epoch": 0.06380857427716849,
	"grad_norm": 0.8022773012436837,
	"learning_rate": 4e-05,
	"loss": 0.2994,
	"num_tokens": 74643231.0,
	"step": 18
	},
	{
	"epoch": 0.06735349507034452,
	"grad_norm": 0.6613390888817,
	"learning_rate": 3.999970528897642e-05,
	"loss": 0.2805,
	"num_tokens": 78824310.0,
	"step": 19
	},
	{
	"epoch": 0.07089841586352055,
	"grad_norm": 0.5922337246617383,
	"learning_rate": 3.999882116555617e-05,
	"loss": 0.2753,
	"num_tokens": 82969886.0,
	"step": 20
	},
	{
	"epoch": 0.07444333665669657,
	"grad_norm": 0.6165272628681612,
	"learning_rate": 3.999734765869049e-05,
	"loss": 0.2653,
	"num_tokens": 87116290.0,
	"step": 21
	},
	{
	"epoch": 0.0779882574498726,
	"grad_norm": 0.5857124781591777,
	"learning_rate": 3.999528481663032e-05,
	"loss": 0.2653,
	"num_tokens": 91270476.0,
	"step": 22
	},
	{
	"epoch": 0.08153317824304863,
	"grad_norm": 0.5551242858067922,
	"learning_rate": 3.999263270692482e-05,
	"loss": 0.2616,
	"num_tokens": 95421666.0,
	"step": 23
	},
	{
	"epoch": 0.08507809903622465,
	"grad_norm": 0.6240117169106899,
	"learning_rate": 3.99893914164191e-05,
	"loss": 0.2472,
	"num_tokens": 99587660.0,
	"step": 24
	},
	{
	"epoch": 0.08862301982940068,
	"grad_norm": 0.3941359772838997,
	"learning_rate": 3.9985561051251374e-05,
	"loss": 0.2417,
	"num_tokens": 103705692.0,
	"step": 25
	},
	{
	"epoch": 0.09216794062257672,
	"grad_norm": 0.4045659562572333,
	"learning_rate": 3.998114173684952e-05,
	"loss": 0.2456,
	"num_tokens": 107856429.0,
	"step": 26
	},
	{
	"epoch": 0.09571286141575275,
	"grad_norm": 0.4866232567967559,
	"learning_rate": 3.997613361792695e-05,
	"loss": 0.2424,
	"num_tokens": 112005235.0,
	"step": 27
	},
	{
	"epoch": 0.09925778220892877,
	"grad_norm": 0.5324580529164159,
	"learning_rate": 3.9970536858477865e-05,
	"loss": 0.2354,
	"num_tokens": 116142105.0,
	"step": 28
	},
	{
	"epoch": 0.1028027030021048,
	"grad_norm": 0.5905422073069264,
	"learning_rate": 3.99643516417719e-05,
	"loss": 0.2395,
	"num_tokens": 120277863.0,
	"step": 29
	},
	{
	"epoch": 0.10634762379528083,
	"grad_norm": 0.3560193746972601,
	"learning_rate": 3.995757817034812e-05,
	"loss": 0.2262,
	"num_tokens": 124445853.0,
	"step": 30
	},
	{
	"epoch": 0.10989254458845685,
	"grad_norm": 0.7479625198109062,
	"learning_rate": 3.995021666600837e-05,
	"loss": 0.2294,
	"num_tokens": 128572905.0,
	"step": 31
	},
	{
	"epoch": 0.11343746538163288,
	"grad_norm": 0.37740288465632527,
	"learning_rate": 3.9942267369810047e-05,
	"loss": 0.2155,
	"num_tokens": 132715922.0,
	"step": 32
	},
	{
	"epoch": 0.11698238617480891,
	"grad_norm": 1.2537732550081822,
	"learning_rate": 3.993373054205816e-05,
	"loss": 0.2271,
	"num_tokens": 136878820.0,
	"step": 33
	},
	{
	"epoch": 0.12052730696798493,
	"grad_norm": 1.3246191780937906,
	"learning_rate": 3.992460646229687e-05,
	"loss": 0.2334,
	"num_tokens": 141003611.0,
	"step": 34
	},
	{
	"epoch": 0.12407222776116096,
	"grad_norm": 0.5475331197892237,
	"learning_rate": 3.991489542930026e-05,
	"loss": 0.226,
	"num_tokens": 145188740.0,
	"step": 35
	},
	{
	"epoch": 0.12761714855433698,
	"grad_norm": 0.9860465849652753,
	"learning_rate": 3.99045977610626e-05,
	"loss": 0.2267,
	"num_tokens": 149334476.0,
	"step": 36
	},
	{
	"epoch": 0.131162069347513,
	"grad_norm": 1.0613237358260414,
	"learning_rate": 3.989371379478795e-05,
	"loss": 0.2271,
	"num_tokens": 153519400.0,
	"step": 37
	},
	{
	"epoch": 0.13470699014068904,
	"grad_norm": 0.6855142763211456,
	"learning_rate": 3.988224388687906e-05,
	"loss": 0.2203,
	"num_tokens": 157701798.0,
	"step": 38
	},
	{
	"epoch": 0.13825191093386507,
	"grad_norm": 0.73798553709363,
	"learning_rate": 3.9870188412925726e-05,
	"loss": 0.2163,
	"num_tokens": 161863690.0,
	"step": 39
	},
	{
	"epoch": 0.1417968317270411,
	"grad_norm": 0.7790134701648027,
	"learning_rate": 3.985754776769253e-05,
	"loss": 0.2149,
	"num_tokens": 166046298.0,
	"step": 40
	},
	{
	"epoch": 0.14534175252021714,
	"grad_norm": 0.631645392825456,
	"learning_rate": 3.984432236510585e-05,
	"loss": 0.2096,
	"num_tokens": 170206941.0,
	"step": 41
	},
	{
	"epoch": 0.14888667331339314,
	"grad_norm": 0.5572867906021802,
	"learning_rate": 3.983051263824035e-05,
	"loss": 0.2105,
	"num_tokens": 174390630.0,
	"step": 42
	},
	{
	"epoch": 0.15243159410656917,
	"grad_norm": 0.5879446535081615,
	"learning_rate": 3.981611903930478e-05,
	"loss": 0.2031,
	"num_tokens": 178560165.0,
	"step": 43
	},
	{
	"epoch": 0.1559765148997452,
	"grad_norm": 0.5002402632592213,
	"learning_rate": 3.9801142039627165e-05,
	"loss": 0.207,
	"num_tokens": 182715582.0,
	"step": 44
	},
	{
	"epoch": 0.15952143569292124,
	"grad_norm": 0.4290924249105735,
	"learning_rate": 3.978558212963939e-05,
	"loss": 0.1982,
	"num_tokens": 186886210.0,
	"step": 45
	},
	{
	"epoch": 0.16306635648609727,
	"grad_norm": 0.42668567805668683,
	"learning_rate": 3.976943981886112e-05,
	"loss": 0.2005,
	"num_tokens": 191062984.0,
	"step": 46
	},
	{
	"epoch": 0.1666112772792733,
	"grad_norm": 0.5362750048048417,
	"learning_rate": 3.975271563588312e-05,
	"loss": 0.2111,
	"num_tokens": 195223724.0,
	"step": 47
	},
	{
	"epoch": 0.1701561980724493,
	"grad_norm": 0.5034815780583302,
	"learning_rate": 3.973541012834997e-05,
	"loss": 0.2056,
	"num_tokens": 199409950.0,
	"step": 48
	},
	{
	"epoch": 0.17370111886562534,
	"grad_norm": 0.3842405776554437,
	"learning_rate": 3.971752386294207e-05,
	"loss": 0.2012,
	"num_tokens": 203581950.0,
	"step": 49
	},
	{
	"epoch": 0.17724603965880137,
	"grad_norm": 0.3795478622590761,
	"learning_rate": 3.9699057425357174e-05,
	"loss": 0.1992,
	"num_tokens": 207746762.0,
	"step": 50
	},
	{
	"epoch": 0.1807909604519774,
	"grad_norm": 0.41196676655946535,
	"learning_rate": 3.968001142029114e-05,
	"loss": 0.1962,
	"num_tokens": 211895377.0,
	"step": 51
	},
	{
	"epoch": 0.18433588124515343,
	"grad_norm": 0.5311374644034718,
	"learning_rate": 3.966038647141813e-05,
	"loss": 0.1961,
	"num_tokens": 216080832.0,
	"step": 52
	},
	{
	"epoch": 0.18788080203832946,
	"grad_norm": 0.3002763055412059,
	"learning_rate": 3.964018322137026e-05,
	"loss": 0.1975,
	"num_tokens": 220248540.0,
	"step": 53
	},
	{
	"epoch": 0.1914257228315055,
	"grad_norm": 0.3992043602305581,
	"learning_rate": 3.9619402331716446e-05,
	"loss": 0.1972,
	"num_tokens": 224410668.0,
	"step": 54
	},
	{
	"epoch": 0.1949706436246815,
	"grad_norm": 0.379124927640763,
	"learning_rate": 3.959804448294084e-05,
	"loss": 0.1924,
	"num_tokens": 228570132.0,
	"step": 55
	},
	{
	"epoch": 0.19851556441785753,
	"grad_norm": 0.5361834702568071,
	"learning_rate": 3.95761103744205e-05,
	"loss": 0.1887,
	"num_tokens": 232742583.0,
	"step": 56
	},
	{
	"epoch": 0.20206048521103356,
	"grad_norm": 0.39137415611156606,
	"learning_rate": 3.9553600724402485e-05,
	"loss": 0.1926,
	"num_tokens": 236914111.0,
	"step": 57
	},
	{
	"epoch": 0.2056054060042096,
	"grad_norm": 0.3303139120776452,
	"learning_rate": 3.953051626998035e-05,
	"loss": 0.1945,
	"num_tokens": 241054452.0,
	"step": 58
	},
	{
	"epoch": 0.20915032679738563,
	"grad_norm": 0.5627639047228225,
	"learning_rate": 3.950685776707001e-05,
	"loss": 0.1916,
	"num_tokens": 245189986.0,
	"step": 59
	},
	{
	"epoch": 0.21269524759056166,
	"grad_norm": 0.3358051848119876,
	"learning_rate": 3.948262599038497e-05,
	"loss": 0.1915,
	"num_tokens": 249352834.0,
	"step": 60
	},
	{
	"epoch": 0.21624016838373766,
	"grad_norm": 0.5199349394926892,
	"learning_rate": 3.9457821733410976e-05,
	"loss": 0.1845,
	"num_tokens": 253519023.0,
	"step": 61
	},
	{
	"epoch": 0.2197850891769137,
	"grad_norm": 0.3826053075090676,
	"learning_rate": 3.943244580838003e-05,
	"loss": 0.1843,
	"num_tokens": 257682498.0,
	"step": 62
	},
	{
	"epoch": 0.22333000997008973,
	"grad_norm": 0.46716468143567286,
	"learning_rate": 3.940649904624377e-05,
	"loss": 0.1929,
	"num_tokens": 261847290.0,
	"step": 63
	},
	{
	"epoch": 0.22687493076326576,
	"grad_norm": 0.44147651684500716,
	"learning_rate": 3.9379982296646296e-05,
	"loss": 0.187,
	"num_tokens": 265980060.0,
	"step": 64
	},
	{
	"epoch": 0.2304198515564418,
	"grad_norm": 0.3780317755164094,
	"learning_rate": 3.935289642789633e-05,
	"loss": 0.1863,
	"num_tokens": 270126113.0,
	"step": 65
	},
	{
	"epoch": 0.23396477234961782,
	"grad_norm": 0.4529072228557033,
	"learning_rate": 3.932524232693875e-05,
	"loss": 0.1843,
	"num_tokens": 274280627.0,
	"step": 66
	},
	{
	"epoch": 0.23750969314279385,
	"grad_norm": 0.44810774110504087,
	"learning_rate": 3.929702089932562e-05,
	"loss": 0.1869,
	"num_tokens": 278445486.0,
	"step": 67
	},
	{
	"epoch": 0.24105461393596986,
	"grad_norm": 0.48082966554229917,
	"learning_rate": 3.926823306918646e-05,
	"loss": 0.1811,
	"num_tokens": 282609287.0,
	"step": 68
	},
	{
	"epoch": 0.2445995347291459,
	"grad_norm": 0.4173541693875608,
	"learning_rate": 3.9238879779198047e-05,
	"loss": 0.1779,
	"num_tokens": 286780812.0,
	"step": 69
	},
	{
	"epoch": 0.24814445552232192,
	"grad_norm": 0.38815021333259625,
	"learning_rate": 3.920896199055351e-05,
	"loss": 0.1893,
	"num_tokens": 290959240.0,
	"step": 70
	},
	{
	"epoch": 0.2516893763154979,
	"grad_norm": 0.4248204944331872,
	"learning_rate": 3.9178480682930854e-05,
	"loss": 0.1776,
	"num_tokens": 295118532.0,
	"step": 71
	},
	{
	"epoch": 0.25523429710867396,
	"grad_norm": 0.4339023197062601,
	"learning_rate": 3.9147436854460887e-05,
	"loss": 0.1733,
	"num_tokens": 299274891.0,
	"step": 72
	},
	{
	"epoch": 0.25877921790185,
	"grad_norm": 0.391013859609384,
	"learning_rate": 3.911583152169457e-05,
	"loss": 0.1824,
	"num_tokens": 303447703.0,
	"step": 73
	},
	{
	"epoch": 0.262324138695026,
	"grad_norm": 0.49700191750412254,
	"learning_rate": 3.908366571956967e-05,
	"loss": 0.1828,
	"num_tokens": 307606763.0,
	"step": 74
	},
	{
	"epoch": 0.26586905948820205,
	"grad_norm": 0.3696540063825826,
	"learning_rate": 3.9050940501376906e-05,
	"loss": 0.1781,
	"num_tokens": 311731778.0,
	"step": 75
	},
	{
	"epoch": 0.2694139802813781,
	"grad_norm": 0.46822551510342364,
	"learning_rate": 3.901765693872545e-05,
	"loss": 0.179,
	"num_tokens": 315897658.0,
	"step": 76
	},
	{
	"epoch": 0.2729589010745541,
	"grad_norm": 0.32801937120334207,
	"learning_rate": 3.898381612150783e-05,
	"loss": 0.1693,
	"num_tokens": 320041471.0,
	"step": 77
	},
	{
	"epoch": 0.27650382186773015,
	"grad_norm": 0.5459549402383508,
	"learning_rate": 3.894941915786427e-05,
	"loss": 0.177,
	"num_tokens": 324176964.0,
	"step": 78
	},
	{
	"epoch": 0.2800487426609062,
	"grad_norm": 0.37030783153703023,
	"learning_rate": 3.891446717414635e-05,
	"loss": 0.1749,
	"num_tokens": 328320045.0,
	"step": 79
	},
	{
	"epoch": 0.2835936634540822,
	"grad_norm": 0.5039219425671296,
	"learning_rate": 3.887896131488019e-05,
	"loss": 0.1812,
	"num_tokens": 332505152.0,
	"step": 80
	},
	{
	"epoch": 0.28713858424725824,
	"grad_norm": 0.5176754138517321,
	"learning_rate": 3.884290274272889e-05,
	"loss": 0.1796,
	"num_tokens": 336679063.0,
	"step": 81
	},
	{
	"epoch": 0.2906835050404343,
	"grad_norm": 0.32115362697178773,
	"learning_rate": 3.880629263845454e-05,
	"loss": 0.1697,
	"num_tokens": 340859771.0,
	"step": 82
	},
	{
	"epoch": 0.2942284258336103,
	"grad_norm": 0.5131941462573518,
	"learning_rate": 3.876913220087952e-05,
	"loss": 0.1726,
	"num_tokens": 345027300.0,
	"step": 83
	},
	{
	"epoch": 0.2977733466267863,
	"grad_norm": 0.3607106960767328,
	"learning_rate": 3.8731422646847213e-05,
	"loss": 0.1699,
	"num_tokens": 349199952.0,
	"step": 84
	},
	{
	"epoch": 0.3013182674199623,
	"grad_norm": 0.4897963518285027,
	"learning_rate": 3.86931652111822e-05,
	"loss": 0.1713,
	"num_tokens": 353366492.0,
	"step": 85
	},
	{
	"epoch": 0.30486318821313835,
	"grad_norm": 0.3775297204433096,
	"learning_rate": 3.865436114664984e-05,
	"loss": 0.1723,
	"num_tokens": 357516015.0,
	"step": 86
	},
	{
	"epoch": 0.3084081090063144,
	"grad_norm": 0.4528205109956923,
	"learning_rate": 3.8615011723915176e-05,
	"loss": 0.1708,
	"num_tokens": 361657506.0,
	"step": 87
	},
	{
	"epoch": 0.3119530297994904,
	"grad_norm": 0.4070025510972201,
	"learning_rate": 3.8575118231501394e-05,
	"loss": 0.1786,
	"num_tokens": 365824807.0,
	"step": 88
	},
	{
	"epoch": 0.31549795059266644,
	"grad_norm": 0.4621421642807574,
	"learning_rate": 3.853468197574762e-05,
	"loss": 0.1666,
	"num_tokens": 369980083.0,
	"step": 89
	},
	{
	"epoch": 0.3190428713858425,
	"grad_norm": 0.4025121388472574,
	"learning_rate": 3.8493704280766084e-05,
	"loss": 0.1667,
	"num_tokens": 374137519.0,
	"step": 90
	},
	{
	"epoch": 0.3225877921790185,
	"grad_norm": 0.4613925558094823,
	"learning_rate": 3.8452186488398865e-05,
	"loss": 0.1731,
	"num_tokens": 378304828.0,
	"step": 91
	},
	{
	"epoch": 0.32613271297219454,
	"grad_norm": 0.4617329301033765,
	"learning_rate": 3.841012995817384e-05,
	"loss": 0.1707,
	"num_tokens": 382483108.0,
	"step": 92
	},
	{
	"epoch": 0.32967763376537057,
	"grad_norm": 0.2695735723641014,
	"learning_rate": 3.836753606726025e-05,
	"loss": 0.1664,
	"num_tokens": 386668761.0,
	"step": 93
	},
	{
	"epoch": 0.3332225545585466,
	"grad_norm": 0.34889174754941027,
	"learning_rate": 3.8324406210423555e-05,
	"loss": 0.1757,
	"num_tokens": 390799475.0,
	"step": 94
	},
	{
	"epoch": 0.33676747535172263,
	"grad_norm": 0.39388355828252747,
	"learning_rate": 3.828074179997978e-05,
	"loss": 0.1649,
	"num_tokens": 394952894.0,
	"step": 95
	},
	{
	"epoch": 0.3403123961448986,
	"grad_norm": 0.5126408077685105,
	"learning_rate": 3.8236544265749275e-05,
	"loss": 0.1782,
	"num_tokens": 399093478.0,
	"step": 96
	},
	{
	"epoch": 0.34385731693807464,
	"grad_norm": 0.30616454492774964,
	"learning_rate": 3.819181505500989e-05,
	"loss": 0.1654,
	"num_tokens": 403264584.0,
	"step": 97
	},
	{
	"epoch": 0.3474022377312507,
	"grad_norm": 0.537865314178385,
	"learning_rate": 3.814655563244954e-05,
	"loss": 0.1765,
	"num_tokens": 407386810.0,
	"step": 98
	},
	{
	"epoch": 0.3509471585244267,
	"grad_norm": 0.3640950988715409,
	"learning_rate": 3.810076748011832e-05,
	"loss": 0.1693,
	"num_tokens": 411529839.0,
	"step": 99
	},
	{
	"epoch": 0.35449207931760274,
	"grad_norm": 0.5371464875193359,
	"learning_rate": 3.805445209737994e-05,
	"loss": 0.167,
	"num_tokens": 415667528.0,
	"step": 100
	},
	{
	"epoch": 0.35803700011077877,
	"grad_norm": 0.39825746056243344,
	"learning_rate": 3.8007611000862586e-05,
	"loss": 0.1668,
	"num_tokens": 419817062.0,
	"step": 101
	},
	{
	"epoch": 0.3615819209039548,
	"grad_norm": 0.5205817796132821,
	"learning_rate": 3.796024572440932e-05,
	"loss": 0.1659,
	"num_tokens": 423978216.0,
	"step": 102
	},
	{
	"epoch": 0.36512684169713083,
	"grad_norm": 0.36426797809101863,
	"learning_rate": 3.7912357819027824e-05,
	"loss": 0.1687,
	"num_tokens": 428147353.0,
	"step": 103
	},
	{
	"epoch": 0.36867176249030686,
	"grad_norm": 0.4869464473778494,
	"learning_rate": 3.786394885283961e-05,
	"loss": 0.1691,
	"num_tokens": 432296888.0,
	"step": 104
	},
	{
	"epoch": 0.3722166832834829,
	"grad_norm": 0.3304420288570703,
	"learning_rate": 3.781502041102867e-05,
	"loss": 0.1626,
	"num_tokens": 436450149.0,
	"step": 105
	},
	{
	"epoch": 0.37576160407665893,
	"grad_norm": 0.4741035939265311,
	"learning_rate": 3.7765574095789586e-05,
	"loss": 0.168,
	"num_tokens": 440582402.0,
	"step": 106
	},
	{
	"epoch": 0.37930652486983496,
	"grad_norm": 0.3898895005976408,
	"learning_rate": 3.771561152627503e-05,
	"loss": 0.1681,
	"num_tokens": 444686547.0,
	"step": 107
	},
	{
	"epoch": 0.382851445663011,
	"grad_norm": 0.44302526207571424,
	"learning_rate": 3.76651343385428e-05,
	"loss": 0.1667,
	"num_tokens": 448853668.0,
	"step": 108
	},
	{
	"epoch": 0.38639636645618697,
	"grad_norm": 0.3994032564412198,
	"learning_rate": 3.761414418550217e-05,
	"loss": 0.1675,
	"num_tokens": 453038587.0,
	"step": 109
	},
	{
	"epoch": 0.389941287249363,
	"grad_norm": 0.3377085234366048,
	"learning_rate": 3.756264273685985e-05,
	"loss": 0.162,
	"num_tokens": 457168405.0,
	"step": 110
	},
	{
	"epoch": 0.39348620804253903,
	"grad_norm": 0.34562853879445016,
	"learning_rate": 3.751063167906523e-05,
	"loss": 0.1586,
	"num_tokens": 461333061.0,
	"step": 111
	},
	{
	"epoch": 0.39703112883571506,
	"grad_norm": 0.2827598722228219,
	"learning_rate": 3.745811271525521e-05,
	"loss": 0.1641,
	"num_tokens": 465474503.0,
	"step": 112
	},
	{
	"epoch": 0.4005760496288911,
	"grad_norm": 0.5022955176024412,
	"learning_rate": 3.740508756519842e-05,
	"loss": 0.1623,
	"num_tokens": 469633427.0,
	"step": 113
	},
	{
	"epoch": 0.4041209704220671,
	"grad_norm": 0.42742919083512276,
	"learning_rate": 3.735155796523888e-05,
	"loss": 0.1607,
	"num_tokens": 473767332.0,
	"step": 114
	},
	{
	"epoch": 0.40766589121524316,
	"grad_norm": 0.29980670769331413,
	"learning_rate": 3.729752566823916e-05,
	"loss": 0.1677,
	"num_tokens": 477918869.0,
	"step": 115
	},
	{
	"epoch": 0.4112108120084192,
	"grad_norm": 0.4625046013553485,
	"learning_rate": 3.7242992443523006e-05,
	"loss": 0.1607,
	"num_tokens": 482045930.0,
	"step": 116
	},
	{
	"epoch": 0.4147557328015952,
	"grad_norm": 0.36894460624816905,
	"learning_rate": 3.7187960076817344e-05,
	"loss": 0.1624,
	"num_tokens": 486204864.0,
	"step": 117
	},
	{
	"epoch": 0.41830065359477125,
	"grad_norm": 0.3181003817832943,
	"learning_rate": 3.7132430370193865e-05,
	"loss": 0.1599,
	"num_tokens": 490349120.0,
	"step": 118
	},
	{
	"epoch": 0.4218455743879473,
	"grad_norm": 0.3804134461885873,
	"learning_rate": 3.707640514200997e-05,
	"loss": 0.1692,
	"num_tokens": 494470835.0,
	"step": 119
	},
	{
	"epoch": 0.4253904951811233,
	"grad_norm": 0.2795183875014872,
	"learning_rate": 3.7019886226849256e-05,
	"loss": 0.1591,
	"num_tokens": 498644300.0,
	"step": 120
	},
	{
	"epoch": 0.42893541597429935,
	"grad_norm": 0.43733574691327487,
	"learning_rate": 3.696287547546143e-05,
	"loss": 0.1571,
	"num_tokens": 502795105.0,
	"step": 121
	},
	{
	"epoch": 0.4324803367674753,
	"grad_norm": 0.3095633493541616,
	"learning_rate": 3.690537475470169e-05,
	"loss": 0.1554,
	"num_tokens": 506934225.0,
	"step": 122
	},
	{
	"epoch": 0.43602525756065136,
	"grad_norm": 0.47417546056909526,
	"learning_rate": 3.684738594746964e-05,
	"loss": 0.1606,
	"num_tokens": 511104184.0,
	"step": 123
	},
	{
	"epoch": 0.4395701783538274,
	"grad_norm": 0.3557202589870836,
	"learning_rate": 3.678891095264758e-05,
	"loss": 0.1566,
	"num_tokens": 515226161.0,
	"step": 124
	},
	{
	"epoch": 0.4431150991470034,
	"grad_norm": 0.3614452668746059,
	"learning_rate": 3.672995168503834e-05,
	"loss": 0.1644,
	"num_tokens": 519392038.0,
	"step": 125
	},
	{
	"epoch": 0.44666001994017945,
	"grad_norm": 0.3261168205931907,
	"learning_rate": 3.6670510075302606e-05,
	"loss": 0.1499,
	"num_tokens": 523558160.0,
	"step": 126
	},
	{
	"epoch": 0.4502049407333555,
	"grad_norm": 0.30948708176247647,
	"learning_rate": 3.6610588069895673e-05,
	"loss": 0.1652,
	"num_tokens": 527738637.0,
	"step": 127
	},
	{
	"epoch": 0.4537498615265315,
	"grad_norm": 0.3511412894964897,
	"learning_rate": 3.655018763100372e-05,
	"loss": 0.154,
	"num_tokens": 531904299.0,
	"step": 128
	},
	{
	"epoch": 0.45729478231970755,
	"grad_norm": 0.3003095848582233,
	"learning_rate": 3.648931073647953e-05,
	"loss": 0.1564,
	"num_tokens": 536070052.0,
	"step": 129
	},
	{
	"epoch": 0.4608397031128836,
	"grad_norm": 0.3585404126205935,
	"learning_rate": 3.6427959379777764e-05,
	"loss": 0.1586,
	"num_tokens": 540190592.0,
	"step": 130
	},
	{
	"epoch": 0.4643846239060596,
	"grad_norm": 0.3489044664861222,
	"learning_rate": 3.636613556988966e-05,
	"loss": 0.1641,
	"num_tokens": 544327379.0,
	"step": 131
	},
	{
	"epoch": 0.46792954469923564,
	"grad_norm": 0.4595810490408498,
	"learning_rate": 3.630384133127726e-05,
	"loss": 0.1557,
	"num_tokens": 548492014.0,
	"step": 132
	},
	{
	"epoch": 0.4714744654924117,
	"grad_norm": 0.24275481188385176,
	"learning_rate": 3.6241078703807076e-05,
	"loss": 0.1542,
	"num_tokens": 552620901.0,
	"step": 133
	},
	{
	"epoch": 0.4750193862855877,
	"grad_norm": 0.4713238402465863,
	"learning_rate": 3.617784974268337e-05,
	"loss": 0.1605,
	"num_tokens": 556772225.0,
	"step": 134
	},
	{
	"epoch": 0.4785643070787637,
	"grad_norm": 0.34581890499888307,
	"learning_rate": 3.6114156518380796e-05,
	"loss": 0.1565,
	"num_tokens": 560889821.0,
	"step": 135
	},
	{
	"epoch": 0.4821092278719397,
	"grad_norm": 0.32264625154197574,
	"learning_rate": 3.6050001116576594e-05,
	"loss": 0.1632,
	"num_tokens": 565048704.0,
	"step": 136
	},
	{
	"epoch": 0.48565414866511575,
	"grad_norm": 0.44078337499729364,
	"learning_rate": 3.598538563808235e-05,
	"loss": 0.1551,
	"num_tokens": 569216407.0,
	"step": 137
	},
	{
	"epoch": 0.4891990694582918,
	"grad_norm": 0.2935856302744625,
	"learning_rate": 3.592031219877514e-05,
	"loss": 0.1554,
	"num_tokens": 573348945.0,
	"step": 138
	},
	{
	"epoch": 0.4927439902514678,
	"grad_norm": 0.3779361197489275,
	"learning_rate": 3.58547829295283e-05,
	"loss": 0.1523,
	"num_tokens": 577528351.0,
	"step": 139
	},
	{
	"epoch": 0.49628891104464384,
	"grad_norm": 0.32149203184662706,
	"learning_rate": 3.578879997614161e-05,
	"loss": 0.155,
	"num_tokens": 581675092.0,
	"step": 140
	},
	{
	"epoch": 0.4998338318378199,
	"grad_norm": 0.3064624867326738,
	"learning_rate": 3.572236549927103e-05,
	"loss": 0.1576,
	"num_tokens": 585827131.0,
	"step": 141
	},
	{
	"epoch": 0.5033787526309959,
	"grad_norm": 0.3326636126268525,
	"learning_rate": 3.5655481674357975e-05,
	"loss": 0.1579,
	"num_tokens": 589961582.0,
	"step": 142
	},
	{
	"epoch": 0.5069236734241719,
	"grad_norm": 0.3324123021514079,
	"learning_rate": 3.558815069155805e-05,
	"loss": 0.1534,
	"num_tokens": 594115447.0,
	"step": 143
	},
	{
	"epoch": 0.5104685942173479,
	"grad_norm": 0.3447751675843534,
	"learning_rate": 3.552037475566936e-05,
	"loss": 0.1527,
	"num_tokens": 598242349.0,
	"step": 144
	},
	{
	"epoch": 0.514013515010524,
	"grad_norm": 0.3742809168987796,
	"learning_rate": 3.545215608606028e-05,
	"loss": 0.1565,
	"num_tokens": 602418761.0,
	"step": 145
	},
	{
	"epoch": 0.5175584358037,
	"grad_norm": 0.30495978863349943,
	"learning_rate": 3.538349691659681e-05,
	"loss": 0.1526,
	"num_tokens": 606591866.0,
	"step": 146
	},
	{
	"epoch": 0.521103356596876,
	"grad_norm": 0.33117731918811877,
	"learning_rate": 3.5314399495569405e-05,
	"loss": 0.1563,
	"num_tokens": 610745695.0,
	"step": 147
	},
	{
	"epoch": 0.524648277390052,
	"grad_norm": 0.3015177181289297,
	"learning_rate": 3.524486608561936e-05,
	"loss": 0.1486,
	"num_tokens": 614880473.0,
	"step": 148
	},
	{
	"epoch": 0.5281931981832281,
	"grad_norm": 0.284649550052416,
	"learning_rate": 3.5174898963664724e-05,
	"loss": 0.1487,
	"num_tokens": 619031559.0,
	"step": 149
	},
	{
	"epoch": 0.5317381189764041,
	"grad_norm": 0.3414570639694516,
	"learning_rate": 3.510450042082573e-05,
	"loss": 0.153,
	"num_tokens": 623156663.0,
	"step": 150
	},
	{
	"epoch": 0.5352830397695801,
	"grad_norm": 0.2981281968741986,
	"learning_rate": 3.5033672762349785e-05,
	"loss": 0.1551,
	"num_tokens": 627294691.0,
	"step": 151
	},
	{
	"epoch": 0.5388279605627562,
	"grad_norm": 0.2629271589477404,
	"learning_rate": 3.496241830753596e-05,
	"loss": 0.1493,
	"num_tokens": 631461674.0,
	"step": 152
	},
	{
	"epoch": 0.5423728813559322,
	"grad_norm": 0.3608253486310783,
	"learning_rate": 3.4890739389659084e-05,
	"loss": 0.1545,
	"num_tokens": 635590751.0,
	"step": 153
	},
	{
	"epoch": 0.5459178021491082,
	"grad_norm": 0.21177523726271086,
	"learning_rate": 3.481863835589328e-05,
	"loss": 0.1553,
	"num_tokens": 639732400.0,
	"step": 154
	},
	{
	"epoch": 0.5494627229422843,
	"grad_norm": 0.42398192528782014,
	"learning_rate": 3.474611756723517e-05,
	"loss": 0.1612,
	"num_tokens": 643882473.0,
	"step": 155
	},
	{
	"epoch": 0.5530076437354603,
	"grad_norm": 0.30107505242043736,
	"learning_rate": 3.4673179398426505e-05,
	"loss": 0.1519,
	"num_tokens": 648065510.0,
	"step": 156
	},
	{
	"epoch": 0.5565525645286363,
	"grad_norm": 0.3043810013809506,
	"learning_rate": 3.459982623787645e-05,
	"loss": 0.1526,
	"num_tokens": 652233323.0,
	"step": 157
	},
	{
	"epoch": 0.5600974853218124,
	"grad_norm": 0.34702231815426027,
	"learning_rate": 3.452606048758332e-05,
	"loss": 0.1565,
	"num_tokens": 656405202.0,
	"step": 158
	},
	{
	"epoch": 0.5636424061149884,
	"grad_norm": 0.28902010952452867,
	"learning_rate": 3.4451884563056e-05,
	"loss": 0.152,
	"num_tokens": 660556935.0,
	"step": 159
	},
	{
	"epoch": 0.5671873269081644,
	"grad_norm": 0.2555340540868128,
	"learning_rate": 3.437730089323477e-05,
	"loss": 0.1482,
	"num_tokens": 664733377.0,
	"step": 160
	},
	{
	"epoch": 0.5707322477013405,
	"grad_norm": 0.38101877807262513,
	"learning_rate": 3.4302311920411834e-05,
	"loss": 0.1526,
	"num_tokens": 668890882.0,
	"step": 161
	},
	{
	"epoch": 0.5742771684945165,
	"grad_norm": 0.24607331672385185,
	"learning_rate": 3.4226920100151276e-05,
	"loss": 0.163,
	"num_tokens": 673046433.0,
	"step": 162
	},
	{
	"epoch": 0.5778220892876925,
	"grad_norm": 0.29643549646460926,
	"learning_rate": 3.415112790120873e-05,
	"loss": 0.1499,
	"num_tokens": 677204163.0,
	"step": 163
	},
	{
	"epoch": 0.5813670100808686,
	"grad_norm": 0.30479825737630384,
	"learning_rate": 3.4074937805450456e-05,
	"loss": 0.1557,
	"num_tokens": 681347986.0,
	"step": 164
	},
	{
	"epoch": 0.5849119308740446,
	"grad_norm": 0.3108324580437024,
	"learning_rate": 3.3998352307772144e-05,
	"loss": 0.1578,
	"num_tokens": 685522877.0,
	"step": 165
	},
	{
	"epoch": 0.5884568516672206,
	"grad_norm": 0.2767583664117791,
	"learning_rate": 3.3921373916017163e-05,
	"loss": 0.1486,
	"num_tokens": 689674502.0,
	"step": 166
	},
	{
	"epoch": 0.5920017724603965,
	"grad_norm": 0.32392919264710807,
	"learning_rate": 3.384400515089449e-05,
	"loss": 0.1517,
	"num_tokens": 693842323.0,
	"step": 167
	},
	{
	"epoch": 0.5955466932535726,
	"grad_norm": 0.24002042689246203,
	"learning_rate": 3.376624854589611e-05,
	"loss": 0.1494,
	"num_tokens": 698009990.0,
	"step": 168
	},
	{
	"epoch": 0.5990916140467486,
	"grad_norm": 0.31818708660075645,
	"learning_rate": 3.36881066472141e-05,
	"loss": 0.1514,
	"num_tokens": 702188457.0,
	"step": 169
	},
	{
	"epoch": 0.6026365348399246,
	"grad_norm": 0.24177379993134826,
	"learning_rate": 3.3609582013657215e-05,
	"loss": 0.1486,
	"num_tokens": 706344756.0,
	"step": 170
	},
	{
	"epoch": 0.6061814556331007,
	"grad_norm": 0.37167432198819556,
	"learning_rate": 3.353067721656716e-05,
	"loss": 0.1477,
	"num_tokens": 710511936.0,
	"step": 171
	},
	{
	"epoch": 0.6097263764262767,
	"grad_norm": 0.37301619449582263,
	"learning_rate": 3.345139483973431e-05,
	"loss": 0.1493,
	"num_tokens": 714668144.0,
	"step": 172
	},
	{
	"epoch": 0.6132712972194527,
	"grad_norm": 0.24272130442480633,
	"learning_rate": 3.3371737479313155e-05,
	"loss": 0.1504,
	"num_tokens": 718834905.0,
	"step": 173
	},
	{
	"epoch": 0.6168162180126288,
	"grad_norm": 0.3932724039309536,
	"learning_rate": 3.329170774373729e-05,
	"loss": 0.1534,
	"num_tokens": 722990366.0,
	"step": 174
	},
	{
	"epoch": 0.6203611388058048,
	"grad_norm": 0.25636827032879894,
	"learning_rate": 3.321130825363396e-05,
	"loss": 0.1451,
	"num_tokens": 727144770.0,
	"step": 175
	},
	{
	"epoch": 0.6239060595989808,
	"grad_norm": 0.4547956542730205,
	"learning_rate": 3.3130541641738286e-05,
	"loss": 0.1494,
	"num_tokens": 731298688.0,
	"step": 176
	},
	{
	"epoch": 0.6274509803921569,
	"grad_norm": 0.3654012748657839,
	"learning_rate": 3.304941055280702e-05,
	"loss": 0.1483,
	"num_tokens": 735446711.0,
	"step": 177
	},
	{
	"epoch": 0.6309959011853329,
	"grad_norm": 0.3849767375032135,
	"learning_rate": 3.2967917643531974e-05,
	"loss": 0.1516,
	"num_tokens": 739616183.0,
	"step": 178
	},
	{
	"epoch": 0.6345408219785089,
	"grad_norm": 0.34385232606199856,
	"learning_rate": 3.2886065582453005e-05,
	"loss": 0.1523,
	"num_tokens": 743788044.0,
	"step": 179
	},
	{
	"epoch": 0.638085742771685,
	"grad_norm": 0.33794251826747923,
	"learning_rate": 3.280385704987063e-05,
	"loss": 0.1438,
	"num_tokens": 747944915.0,
	"step": 180
	},
	{
	"epoch": 0.641630663564861,
	"grad_norm": 0.25481833529296,
	"learning_rate": 3.272129473775828e-05,
	"loss": 0.1481,
	"num_tokens": 752112368.0,
	"step": 181
	},
	{
	"epoch": 0.645175584358037,
	"grad_norm": 0.355284556650196,
	"learning_rate": 3.263838134967412e-05,
	"loss": 0.1499,
	"num_tokens": 756236925.0,
	"step": 182
	},
	{
	"epoch": 0.648720505151213,
	"grad_norm": 0.24751521316236808,
	"learning_rate": 3.255511960067252e-05,
	"loss": 0.1481,
	"num_tokens": 760343199.0,
	"step": 183
	},
	{
	"epoch": 0.6522654259443891,
	"grad_norm": 0.41064894722400624,
	"learning_rate": 3.2471512217215204e-05,
	"loss": 0.1508,
	"num_tokens": 764485741.0,
	"step": 184
	},
	{
	"epoch": 0.6558103467375651,
	"grad_norm": 0.32963773341655855,
	"learning_rate": 3.238756193708188e-05,
	"loss": 0.1464,
	"num_tokens": 768595407.0,
	"step": 185
	},
	{
	"epoch": 0.6593552675307411,
	"grad_norm": 0.3598952607395606,
	"learning_rate": 3.230327150928067e-05,
	"loss": 0.1487,
	"num_tokens": 772764248.0,
	"step": 186
	},
	{
	"epoch": 0.6629001883239172,
	"grad_norm": 0.3605219687084764,
	"learning_rate": 3.2218643693958035e-05,
	"loss": 0.1529,
	"num_tokens": 776927254.0,
	"step": 187
	},
	{
	"epoch": 0.6664451091170932,
	"grad_norm": 0.28467364943800905,
	"learning_rate": 3.2133681262308447e-05,
	"loss": 0.1498,
	"num_tokens": 781098016.0,
	"step": 188
	},
	{
	"epoch": 0.6699900299102692,
	"grad_norm": 0.30241851287915467,
	"learning_rate": 3.204838699648357e-05,
	"loss": 0.1455,
	"num_tokens": 785276914.0,
	"step": 189
	},
	{
	"epoch": 0.6735349507034453,
	"grad_norm": 0.2465764898565235,
	"learning_rate": 3.1962763689501235e-05,
	"loss": 0.1508,
	"num_tokens": 789442048.0,
	"step": 190
	},
	{
	"epoch": 0.6770798714966213,
	"grad_norm": 0.29766972455864044,
	"learning_rate": 3.1876814145153954e-05,
	"loss": 0.148,
	"num_tokens": 793580696.0,
	"step": 191
	},
	{
	"epoch": 0.6806247922897972,
	"grad_norm": 0.24531110399233266,
	"learning_rate": 3.179054117791705e-05,
	"loss": 0.1465,
	"num_tokens": 797721951.0,
	"step": 192
	},
	{
	"epoch": 0.6841697130829733,
	"grad_norm": 0.3543604079511646,
	"learning_rate": 3.1703947612856596e-05,
	"loss": 0.1456,
	"num_tokens": 801841146.0,
	"step": 193
	},
	{
	"epoch": 0.6877146338761493,
	"grad_norm": 0.2868052445806406,
	"learning_rate": 3.161703628553683e-05,
	"loss": 0.1485,
	"num_tokens": 805999155.0,
	"step": 194
	},
	{
	"epoch": 0.6912595546693253,
	"grad_norm": 0.23393926164866768,
	"learning_rate": 3.1529810041927336e-05,
	"loss": 0.1371,
	"num_tokens": 810149998.0,
	"step": 195
	},
	{
	"epoch": 0.6948044754625013,
	"grad_norm": 0.28612015053595774,
	"learning_rate": 3.144227173830985e-05,
	"loss": 0.1477,
	"num_tokens": 814330921.0,
	"step": 196
	},
	{
	"epoch": 0.6983493962556774,
	"grad_norm": 0.2542200263121306,
	"learning_rate": 3.135442424118468e-05,
	"loss": 0.1461,
	"num_tokens": 818490661.0,
	"step": 197
	},
	{
	"epoch": 0.7018943170488534,
	"grad_norm": 0.2571089373419467,
	"learning_rate": 3.126627042717696e-05,
	"loss": 0.1393,
	"num_tokens": 822650647.0,
	"step": 198
	},
	{
	"epoch": 0.7054392378420294,
	"grad_norm": 0.25492137839151535,
	"learning_rate": 3.117781318294229e-05,
	"loss": 0.1461,
	"num_tokens": 826808135.0,
	"step": 199
	},
	{
	"epoch": 0.7089841586352055,
	"grad_norm": 0.34271487486223645,
	"learning_rate": 3.108905540507237e-05,
	"loss": 0.1442,
	"num_tokens": 830971239.0,
	"step": 200
	},
	{
	"epoch": 0.7125290794283815,
	"grad_norm": 0.304177005018979,
	"learning_rate": 3.1e-05,
	"loss": 0.1435,
	"num_tokens": 835108431.0,
	"step": 201
	},
	{
	"epoch": 0.7160740002215575,
	"grad_norm": 0.2390623733934396,
	"learning_rate": 3.0910649883904064e-05,
	"loss": 0.1465,
	"num_tokens": 839259440.0,
	"step": 202
	},
	{
	"epoch": 0.7196189210147336,
	"grad_norm": 0.29840406003663317,
	"learning_rate": 3.082100798261389e-05,
	"loss": 0.1445,
	"num_tokens": 843397078.0,
	"step": 203
	},
	{
	"epoch": 0.7231638418079096,
	"grad_norm": 0.22969058279101331,
	"learning_rate": 3.073107723151354e-05,
	"loss": 0.1441,
	"num_tokens": 847563677.0,
	"step": 204
	},
	{
	"epoch": 0.7267087626010856,
	"grad_norm": 0.3272563188086576,
	"learning_rate": 3.064086057544564e-05,
	"loss": 0.1512,
	"num_tokens": 851722163.0,
	"step": 205
	},
	{
	"epoch": 0.7302536833942617,
	"grad_norm": 0.21512474216808491,
	"learning_rate": 3.055036096861498e-05,
	"loss": 0.1439,
	"num_tokens": 855875958.0,
	"step": 206
	},
	{
	"epoch": 0.7337986041874377,
	"grad_norm": 0.37548880774430815,
	"learning_rate": 3.0459581374491765e-05,
	"loss": 0.1454,
	"num_tokens": 860035379.0,
	"step": 207
	},
	{
	"epoch": 0.7373435249806137,
	"grad_norm": 0.32309417233636023,
	"learning_rate": 3.036852476571455e-05,
	"loss": 0.1417,
	"num_tokens": 864221682.0,
	"step": 208
	},
	{
	"epoch": 0.7408884457737898,
	"grad_norm": 0.2639797506747037,
	"learning_rate": 3.0277194123992934e-05,
	"loss": 0.141,
	"num_tokens": 868351941.0,
	"step": 209
	},
	{
	"epoch": 0.7444333665669658,
	"grad_norm": 0.3708319727479278,
	"learning_rate": 3.018559244000991e-05,
	"loss": 0.1439,
	"num_tokens": 872479823.0,
	"step": 210
	},
	{
	"epoch": 0.7479782873601418,
	"grad_norm": 0.28491097274273697,
	"learning_rate": 3.0093722713323924e-05,
	"loss": 0.1395,
	"num_tokens": 876634490.0,
	"step": 211
	},
	{
	"epoch": 0.7515232081533179,
	"grad_norm": 0.30889497790140963,
	"learning_rate": 3.0001587952270664e-05,
	"loss": 0.1468,
	"num_tokens": 880783148.0,
	"step": 212
	},
	{
	"epoch": 0.7550681289464939,
	"grad_norm": 0.32531761176278,
	"learning_rate": 2.9909191173864548e-05,
	"loss": 0.1449,
	"num_tokens": 884940118.0,
	"step": 213
	},
	{
	"epoch": 0.7586130497396699,
	"grad_norm": 0.20279522867012,
	"learning_rate": 2.9816535403699932e-05,
	"loss": 0.145,
	"num_tokens": 889109395.0,
	"step": 214
	},
	{
	"epoch": 0.762157970532846,
	"grad_norm": 0.3374184836945865,
	"learning_rate": 2.9723623675852007e-05,
	"loss": 0.1479,
	"num_tokens": 893271571.0,
	"step": 215
	},
	{
	"epoch": 0.765702891326022,
	"grad_norm": 0.2542704763336209,
	"learning_rate": 2.96304590327775e-05,
	"loss": 0.144,
	"num_tokens": 897393895.0,
	"step": 216
	},
	{
	"epoch": 0.769247812119198,
	"grad_norm": 0.32803351326153607,
	"learning_rate": 2.953704452521499e-05,
	"loss": 0.1399,
	"num_tokens": 901515472.0,
	"step": 217
	},
	{
	"epoch": 0.7727927329123739,
	"grad_norm": 0.27818416571805094,
	"learning_rate": 2.944338321208506e-05,
	"loss": 0.1463,
	"num_tokens": 905703316.0,
	"step": 218
	},
	{
	"epoch": 0.77633765370555,
	"grad_norm": 0.2634072464657986,
	"learning_rate": 2.9349478160390084e-05,
	"loss": 0.1436,
	"num_tokens": 909873568.0,
	"step": 219
	},
	{
	"epoch": 0.779882574498726,
	"grad_norm": 0.2308292613459674,
	"learning_rate": 2.925533244511384e-05,
	"loss": 0.1401,
	"num_tokens": 914054923.0,
	"step": 220
	},
	{
	"epoch": 0.783427495291902,
	"grad_norm": 0.22367750290995544,
	"learning_rate": 2.9160949149120775e-05,
	"loss": 0.142,
	"num_tokens": 918218164.0,
	"step": 221
	},
	{
	"epoch": 0.7869724160850781,
	"grad_norm": 0.2520505219139085,
	"learning_rate": 2.9066331363055097e-05,
	"loss": 0.1428,
	"num_tokens": 922371370.0,
	"step": 222
	},
	{
	"epoch": 0.7905173368782541,
	"grad_norm": 0.27272722199565863,
	"learning_rate": 2.8971482185239525e-05,
	"loss": 0.1417,
	"num_tokens": 926525487.0,
	"step": 223
	},
	{
	"epoch": 0.7940622576714301,
	"grad_norm": 0.20259528764075876,
	"learning_rate": 2.8876404721573877e-05,
	"loss": 0.1401,
	"num_tokens": 930663641.0,
	"step": 224
	},
	{
	"epoch": 0.7976071784646062,
	"grad_norm": 0.3662174371055876,
	"learning_rate": 2.8781102085433336e-05,
	"loss": 0.1415,
	"num_tokens": 934820759.0,
	"step": 225
	},
	{
	"epoch": 0.8011520992577822,
	"grad_norm": 0.2337578727476228,
	"learning_rate": 2.8685577397566497e-05,
	"loss": 0.1416,
	"num_tokens": 938960601.0,
	"step": 226
	},
	{
	"epoch": 0.8046970200509582,
	"grad_norm": 0.40948316434282594,
	"learning_rate": 2.8589833785993214e-05,
	"loss": 0.1411,
	"num_tokens": 943137774.0,
	"step": 227
	},
	{
	"epoch": 0.8082419408441343,
	"grad_norm": 0.2555067751918292,
	"learning_rate": 2.8493874385902103e-05,
	"loss": 0.1425,
	"num_tokens": 947308465.0,
	"step": 228
	},
	{
	"epoch": 0.8117868616373103,
	"grad_norm": 0.39307861166618363,
	"learning_rate": 2.839770233954797e-05,
	"loss": 0.1416,
	"num_tokens": 951468143.0,
	"step": 229
	},
	{
	"epoch": 0.8153317824304863,
	"grad_norm": 0.30765093297180846,
	"learning_rate": 2.830132079614882e-05,
	"loss": 0.1472,
	"num_tokens": 955628321.0,
	"step": 230
	},
	{
	"epoch": 0.8188767032236623,
	"grad_norm": 0.2927981288584913,
	"learning_rate": 2.820473291178281e-05,
	"loss": 0.1413,
	"num_tokens": 959736918.0,
	"step": 231
	},
	{
	"epoch": 0.8224216240168384,
	"grad_norm": 0.25278970509215726,
	"learning_rate": 2.8107941849284857e-05,
	"loss": 0.1458,
	"num_tokens": 963922595.0,
	"step": 232
	},
	{
	"epoch": 0.8259665448100144,
	"grad_norm": 0.316613976895548,
	"learning_rate": 2.8010950778143083e-05,
	"loss": 0.1422,
	"num_tokens": 968089918.0,
	"step": 233
	},
	{
	"epoch": 0.8295114656031904,
	"grad_norm": 0.2533338291444822,
	"learning_rate": 2.791376287439503e-05,
	"loss": 0.1472,
	"num_tokens": 972267121.0,
	"step": 234
	},
	{
	"epoch": 0.8330563863963665,
	"grad_norm": 0.3099689918014671,
	"learning_rate": 2.7816381320523645e-05,
	"loss": 0.1393,
	"num_tokens": 976375175.0,
	"step": 235
	},
	{
	"epoch": 0.8366013071895425,
	"grad_norm": 0.23776634820165252,
	"learning_rate": 2.7718809305353088e-05,
	"loss": 0.1372,
	"num_tokens": 980544700.0,
	"step": 236
	},
	{
	"epoch": 0.8401462279827185,
	"grad_norm": 0.3104817869126944,
	"learning_rate": 2.7621050023944306e-05,
	"loss": 0.1519,
	"num_tokens": 984711796.0,
	"step": 237
	},
	{
	"epoch": 0.8436911487758946,
	"grad_norm": 0.2712243166267924,
	"learning_rate": 2.7523106677490396e-05,
	"loss": 0.1473,
	"num_tokens": 988868382.0,
	"step": 238
	},
	{
	"epoch": 0.8472360695690706,
	"grad_norm": 0.2646601645211145,
	"learning_rate": 2.7424982473211778e-05,
	"loss": 0.1426,
	"num_tokens": 993047823.0,
	"step": 239
	},
	{
	"epoch": 0.8507809903622466,
	"grad_norm": 0.25855223070160166,
	"learning_rate": 2.7326680624251206e-05,
	"loss": 0.1409,
	"num_tokens": 997198789.0,
	"step": 240
	},
	{
	"epoch": 0.8543259111554227,
	"grad_norm": 0.22155591786968054,
	"learning_rate": 2.7228204349568524e-05,
	"loss": 0.1384,
	"num_tokens": 1001350071.0,
	"step": 241
	},
	{
	"epoch": 0.8578708319485987,
	"grad_norm": 0.2698859361188432,
	"learning_rate": 2.7129556873835236e-05,
	"loss": 0.1409,
	"num_tokens": 1005514700.0,
	"step": 242
	},
	{
	"epoch": 0.8614157527417746,
	"grad_norm": 0.24498647089532183,
	"learning_rate": 2.7030741427328973e-05,
	"loss": 0.1357,
	"num_tokens": 1009659956.0,
	"step": 243
	},
	{
	"epoch": 0.8649606735349507,
	"grad_norm": 0.2753668036642962,
	"learning_rate": 2.693176124582766e-05,
	"loss": 0.1375,
	"num_tokens": 1013800817.0,
	"step": 244
	},
	{
	"epoch": 0.8685055943281267,
	"grad_norm": 0.2638077074892478,
	"learning_rate": 2.6832619570503585e-05,
	"loss": 0.1393,
	"num_tokens": 1017950749.0,
	"step": 245
	},
	{
	"epoch": 0.8720505151213027,
	"grad_norm": 0.23568994377455932,
	"learning_rate": 2.6733319647817265e-05,
	"loss": 0.1338,
	"num_tokens": 1022082539.0,
	"step": 246
	},
	{
	"epoch": 0.8755954359144787,
	"grad_norm": 0.1808848127544263,
	"learning_rate": 2.6633864729411137e-05,
	"loss": 0.1406,
	"num_tokens": 1026267922.0,
	"step": 247
	},
	{
	"epoch": 0.8791403567076548,
	"grad_norm": 0.243246405971141,
	"learning_rate": 2.6534258072003045e-05,
	"loss": 0.1327,
	"num_tokens": 1030452020.0,
	"step": 248
	},
	{
	"epoch": 0.8826852775008308,
	"grad_norm": 0.29965038976730146,
	"learning_rate": 2.6434502937279667e-05,
	"loss": 0.141,
	"num_tokens": 1034632579.0,
	"step": 249
	},
	{
	"epoch": 0.8862301982940068,
	"grad_norm": 0.17800987518961833,
	"learning_rate": 2.6334602591789654e-05,
	"loss": 0.1331,
	"num_tokens": 1038779178.0,
	"step": 250
	},
	{
	"epoch": 0.8897751190871829,
	"grad_norm": 0.23709748267523076,
	"learning_rate": 2.6234560306836684e-05,
	"loss": 0.1398,
	"num_tokens": 1042950968.0,
	"step": 251
	},
	{
	"epoch": 0.8933200398803589,
	"grad_norm": 0.30291892142510585,
	"learning_rate": 2.6134379358372326e-05,
	"loss": 0.1398,
	"num_tokens": 1047119848.0,
	"step": 252
	},
	{
	"epoch": 0.8968649606735349,
	"grad_norm": 0.16152592028734267,
	"learning_rate": 2.6034063026888787e-05,
	"loss": 0.1403,
	"num_tokens": 1051283969.0,
	"step": 253
	},
	{
	"epoch": 0.900409881466711,
	"grad_norm": 0.25444769330544037,
	"learning_rate": 2.593361459731149e-05,
	"loss": 0.1355,
	"num_tokens": 1055425597.0,
	"step": 254
	},
	{
	"epoch": 0.903954802259887,
	"grad_norm": 0.27983021748969955,
	"learning_rate": 2.5833037358891484e-05,
	"loss": 0.1401,
	"num_tokens": 1059587976.0,
	"step": 255
	},
	{
	"epoch": 0.907499723053063,
	"grad_norm": 0.16712177351868332,
	"learning_rate": 2.5732334605097763e-05,
	"loss": 0.1414,
	"num_tokens": 1063741966.0,
	"step": 256
	},
	{
	"epoch": 0.9110446438462391,
	"grad_norm": 0.2914241162205053,
	"learning_rate": 2.563150963350939e-05,
	"loss": 0.1386,
	"num_tokens": 1067868051.0,
	"step": 257
	},
	{
	"epoch": 0.9145895646394151,
	"grad_norm": 0.2382333201703009,
	"learning_rate": 2.553056574570755e-05,
	"loss": 0.1399,
	"num_tokens": 1072025546.0,
	"step": 258
	},
	{
	"epoch": 0.9181344854325911,
	"grad_norm": 0.17192324762859762,
	"learning_rate": 2.5429506247167396e-05,
	"loss": 0.1404,
	"num_tokens": 1076186744.0,
	"step": 259
	},
	{
	"epoch": 0.9216794062257672,
	"grad_norm": 0.22468943659976826,
	"learning_rate": 2.532833444714986e-05,
	"loss": 0.1376,
	"num_tokens": 1080365208.0,
	"step": 260
	},
	{
	"epoch": 0.9252243270189432,
	"grad_norm": 0.25737129816327253,
	"learning_rate": 2.5227053658593245e-05,
	"loss": 0.1378,
	"num_tokens": 1084538417.0,
	"step": 261
	},
	{
	"epoch": 0.9287692478121192,
	"grad_norm": 0.1874221453930141,
	"learning_rate": 2.512566719800475e-05,
	"loss": 0.1336,
	"num_tokens": 1088695876.0,
	"step": 262
	},
	{
	"epoch": 0.9323141686052953,
	"grad_norm": 0.2260305170859545,
	"learning_rate": 2.5024178385351886e-05,
	"loss": 0.1398,
	"num_tokens": 1092848235.0,
	"step": 263
	},
	{
	"epoch": 0.9358590893984713,
	"grad_norm": 0.21226305209473478,
	"learning_rate": 2.4922590543953747e-05,
	"loss": 0.1326,
	"num_tokens": 1097008418.0,
	"step": 264
	},
	{
	"epoch": 0.9394040101916473,
	"grad_norm": 0.21373794790168707,
	"learning_rate": 2.4820907000372204e-05,
	"loss": 0.1354,
	"num_tokens": 1101170297.0,
	"step": 265
	},
	{
	"epoch": 0.9429489309848234,
	"grad_norm": 0.263478352475459,
	"learning_rate": 2.4719131084302923e-05,
	"loss": 0.1387,
	"num_tokens": 1105317528.0,
	"step": 266
	},
	{
	"epoch": 0.9464938517779994,
	"grad_norm": 0.2277374524901536,
	"learning_rate": 2.461726612846642e-05,
	"loss": 0.1388,
	"num_tokens": 1109492418.0,
	"step": 267
	},
	{
	"epoch": 0.9500387725711754,
	"grad_norm": 0.2187246927355263,
	"learning_rate": 2.4515315468498836e-05,
	"loss": 0.1393,
	"num_tokens": 1113629137.0,
	"step": 268
	},
	{
	"epoch": 0.9535836933643513,
	"grad_norm": 0.1671317213668108,
	"learning_rate": 2.441328244284276e-05,
	"loss": 0.1422,
	"num_tokens": 1117772392.0,
	"step": 269
	},
	{
	"epoch": 0.9571286141575274,
	"grad_norm": 0.18605950569051266,
	"learning_rate": 2.4311170392637918e-05,
	"loss": 0.1371,
	"num_tokens": 1121932842.0,
	"step": 270
	},
	{
	"epoch": 0.9606735349507034,
	"grad_norm": 0.23546192314395067,
	"learning_rate": 2.4208982661611735e-05,
	"loss": 0.1351,
	"num_tokens": 1126090284.0,
	"step": 271
	},
	{
	"epoch": 0.9642184557438794,
	"grad_norm": 0.19348662559792806,
	"learning_rate": 2.4106722595969855e-05,
	"loss": 0.137,
	"num_tokens": 1130235713.0,
	"step": 272
	},
	{
	"epoch": 0.9677633765370555,
	"grad_norm": 0.25187215659418233,
	"learning_rate": 2.400439354428656e-05,
	"loss": 0.1391,
	"num_tokens": 1134376662.0,
	"step": 273
	},
	{
	"epoch": 0.9713082973302315,
	"grad_norm": 0.21541285356841056,
	"learning_rate": 2.3901998857395152e-05,
	"loss": 0.1367,
	"num_tokens": 1138544531.0,
	"step": 274
	},
	{
	"epoch": 0.9748532181234075,
	"grad_norm": 0.17279449330441113,
	"learning_rate": 2.3799541888278167e-05,
	"loss": 0.1354,
	"num_tokens": 1142693953.0,
	"step": 275
	},
	{
	"epoch": 0.9783981389165836,
	"grad_norm": 0.16096358277689357,
	"learning_rate": 2.3697025991957644e-05,
	"loss": 0.1379,
	"num_tokens": 1146849181.0,
	"step": 276
	},
	{
	"epoch": 0.9819430597097596,
	"grad_norm": 0.1719727689378624,
	"learning_rate": 2.3594454525385216e-05,
	"loss": 0.1326,
	"num_tokens": 1150988214.0,
	"step": 277
	},
	{
	"epoch": 0.9854879805029356,
	"grad_norm": 0.19407521199513725,
	"learning_rate": 2.3491830847332204e-05,
	"loss": 0.1421,
	"num_tokens": 1155143892.0,
	"step": 278
	},
	{
	"epoch": 0.9890329012961117,
	"grad_norm": 0.24815932582795064,
	"learning_rate": 2.338915831827964e-05,
	"loss": 0.133,
	"num_tokens": 1159286342.0,
	"step": 279
	},
	{
	"epoch": 0.9925778220892877,
	"grad_norm": 0.20933724273988658,
	"learning_rate": 2.3286440300308187e-05,
	"loss": 0.133,
	"num_tokens": 1163427753.0,
	"step": 280
	},
	{
	"epoch": 0.9961227428824637,
	"grad_norm": 0.2202384251935729,
	"learning_rate": 2.3183680156988112e-05,
	"loss": 0.1356,
	"num_tokens": 1167576925.0,
	"step": 281
	},
	{
	"epoch": 0.9996676636756398,
	"grad_norm": 0.19246276851979996,
	"learning_rate": 2.3080881253269074e-05,
	"loss": 0.1353,
	"num_tokens": 1171744045.0,
	"step": 282
	},
	{
	"epoch": 1.0,
	"grad_norm": 0.2719118710677436,
	"learning_rate": 2.2978046955369977e-05,
	"loss": 0.1515,
	"num_tokens": 1171940294.0,
	"step": 283
	},
	{
	"epoch": 1.003544920793176,
	"grad_norm": 0.20803104412993495,
	"learning_rate": 2.2875180630668728e-05,
	"loss": 0.1278,
	"num_tokens": 1176079293.0,
	"step": 284
	},
	{
	"epoch": 1.007089841586352,
	"grad_norm": 0.3617203857230133,
	"learning_rate": 2.2772285647591983e-05,
	"loss": 0.1315,
	"num_tokens": 1180224341.0,
	"step": 285
	},
	{
	"epoch": 1.010634762379528,
	"grad_norm": 0.21444969618279658,
	"learning_rate": 2.2669365375504824e-05,
	"loss": 0.1306,
	"num_tokens": 1184376208.0,
	"step": 286
	},
	{
	"epoch": 1.0141796831727041,
	"grad_norm": 0.3243988978749438,
	"learning_rate": 2.256642318460045e-05,
	"loss": 0.1289,
	"num_tokens": 1188519148.0,
	"step": 287
	},
	{
	"epoch": 1.01772460396588,
	"grad_norm": 0.2510005019241661,
	"learning_rate": 2.24634624457898e-05,
	"loss": 0.1348,
	"num_tokens": 1192686260.0,
	"step": 288
	},
	{
	"epoch": 1.0212695247590562,
	"grad_norm": 0.267185107283224,
	"learning_rate": 2.2360486530591164e-05,
	"loss": 0.13,
	"num_tokens": 1196795971.0,
	"step": 289
	},
	{
	"epoch": 1.0248144455522321,
	"grad_norm": 0.24029317008837686,
	"learning_rate": 2.2257498811019823e-05,
	"loss": 0.1273,
	"num_tokens": 1200923021.0,
	"step": 290
	},
	{
	"epoch": 1.0283593663454083,
	"grad_norm": 0.24220294423644384,
	"learning_rate": 2.215450265947757e-05,
	"loss": 0.128,
	"num_tokens": 1205048465.0,
	"step": 291
	},
	{
	"epoch": 1.0319042871385842,
	"grad_norm": 0.2092040702212028,
	"learning_rate": 2.205150144864233e-05,
	"loss": 0.1299,
	"num_tokens": 1209200516.0,
	"step": 292
	},
	{
	"epoch": 1.0354492079317603,
	"grad_norm": 0.21380979555107005,
	"learning_rate": 2.1948498551357682e-05,
	"loss": 0.1335,
	"num_tokens": 1213378570.0,
	"step": 293
	},
	{
	"epoch": 1.0389941287249362,
	"grad_norm": 0.2548628555393858,
	"learning_rate": 2.184549734052244e-05,
	"loss": 0.1325,
	"num_tokens": 1217533001.0,
	"step": 294
	},
	{
	"epoch": 1.0425390495181124,
	"grad_norm": 0.19817297458615032,
	"learning_rate": 2.174250118898018e-05,
	"loss": 0.1286,
	"num_tokens": 1221663872.0,
	"step": 295
	},
	{
	"epoch": 1.0460839703112883,
	"grad_norm": 0.22780787576520717,
	"learning_rate": 2.163951346940884e-05,
	"loss": 0.127,
	"num_tokens": 1225809625.0,
	"step": 296
	},
	{
	"epoch": 1.0496288911044644,
	"grad_norm": 0.24565352893491163,
	"learning_rate": 2.153653755421021e-05,
	"loss": 0.1208,
	"num_tokens": 1229966468.0,
	"step": 297
	},
	{
	"epoch": 1.0531738118976404,
	"grad_norm": 0.19753560104717474,
	"learning_rate": 2.1433576815399552e-05,
	"loss": 0.1252,
	"num_tokens": 1234066283.0,
	"step": 298
	},
	{
	"epoch": 1.0567187326908165,
	"grad_norm": 0.22910270958785872,
	"learning_rate": 2.1330634624495185e-05,
	"loss": 0.1278,
	"num_tokens": 1238219836.0,
	"step": 299
	},
	{
	"epoch": 1.0602636534839924,
	"grad_norm": 0.15310937281457898,
	"learning_rate": 2.122771435240802e-05,
	"loss": 0.127,
	"num_tokens": 1242404520.0,
	"step": 300
	},
	{
	"epoch": 1.0638085742771686,
	"grad_norm": 0.22265360583073315,
	"learning_rate": 2.1124819369331275e-05,
	"loss": 0.1255,
	"num_tokens": 1246566734.0,
	"step": 301
	},
	{
	"epoch": 1.0673534950703445,
	"grad_norm": 0.18062775429858302,
	"learning_rate": 2.1021953044630035e-05,
	"loss": 0.1239,
	"num_tokens": 1250713435.0,
	"step": 302
	},
	{
	"epoch": 1.0708984158635206,
	"grad_norm": 0.13049104778765483,
	"learning_rate": 2.0919118746730935e-05,
	"loss": 0.1254,
	"num_tokens": 1254873129.0,
	"step": 303
	},
	{
	"epoch": 1.0744433366566966,
	"grad_norm": 0.1538963764904334,
	"learning_rate": 2.08163198430119e-05,
	"loss": 0.1296,
	"num_tokens": 1259042683.0,
	"step": 304
	},
	{
	"epoch": 1.0779882574498727,
	"grad_norm": 0.14904869848403005,
	"learning_rate": 2.0713559699691816e-05,
	"loss": 0.1269,
	"num_tokens": 1263199926.0,
	"step": 305
	},
	{
	"epoch": 1.0815331782430486,
	"grad_norm": 0.13569002367997077,
	"learning_rate": 2.0610841681720368e-05,
	"loss": 0.1264,
	"num_tokens": 1267337582.0,
	"step": 306
	},
	{
	"epoch": 1.0850780990362248,
	"grad_norm": 0.15850267239729662,
	"learning_rate": 2.0508169152667802e-05,
	"loss": 0.1227,
	"num_tokens": 1271517117.0,
	"step": 307
	},
	{
	"epoch": 1.0886230198294007,
	"grad_norm": 0.16641369280363377,
	"learning_rate": 2.0405545474614793e-05,
	"loss": 0.125,
	"num_tokens": 1275665727.0,
	"step": 308
	},
	{
	"epoch": 1.0921679406225766,
	"grad_norm": 0.14317135663168587,
	"learning_rate": 2.0302974008042362e-05,
	"loss": 0.1258,
	"num_tokens": 1279817792.0,
	"step": 309
	},
	{
	"epoch": 1.0957128614157527,
	"grad_norm": 0.14619180431504442,
	"learning_rate": 2.0200458111721842e-05,
	"loss": 0.1251,
	"num_tokens": 1283965620.0,
	"step": 310
	},
	{
	"epoch": 1.0992577822089287,
	"grad_norm": 0.12647299931880404,
	"learning_rate": 2.0098001142604853e-05,
	"loss": 0.1219,
	"num_tokens": 1288143398.0,
	"step": 311
	},
	{
	"epoch": 1.1028027030021048,
	"grad_norm": 0.1511073236834039,
	"learning_rate": 1.9995606455713445e-05,
	"loss": 0.1232,
	"num_tokens": 1292316366.0,
	"step": 312
	},
	{
	"epoch": 1.1063476237952807,
	"grad_norm": 0.1627253998908657,
	"learning_rate": 1.9893277404030157e-05,
	"loss": 0.1256,
	"num_tokens": 1296455111.0,
	"step": 313
	},
	{
	"epoch": 1.1098925445884569,
	"grad_norm": 0.19928286619457392,
	"learning_rate": 1.979101733838827e-05,
	"loss": 0.1261,
	"num_tokens": 1300628644.0,
	"step": 314
	},
	{
	"epoch": 1.1134374653816328,
	"grad_norm": 0.1650647582456179,
	"learning_rate": 1.9688829607362084e-05,
	"loss": 0.1281,
	"num_tokens": 1304771653.0,
	"step": 315
	},
	{
	"epoch": 1.116982386174809,
	"grad_norm": 0.14134540550562566,
	"learning_rate": 1.9586717557157242e-05,
	"loss": 0.1238,
	"num_tokens": 1308956169.0,
	"step": 316
	},
	{
	"epoch": 1.1205273069679849,
	"grad_norm": 0.17803934425160586,
	"learning_rate": 1.9484684531501173e-05,
	"loss": 0.1273,
	"num_tokens": 1313097080.0,
	"step": 317
	},
	{
	"epoch": 1.124072227761161,
	"grad_norm": 0.19397851252453754,
	"learning_rate": 1.9382733871533586e-05,
	"loss": 0.1236,
	"num_tokens": 1317247032.0,
	"step": 318
	},
	{
	"epoch": 1.127617148554337,
	"grad_norm": 0.18547486566417304,
	"learning_rate": 1.928086891569708e-05,
	"loss": 0.121,
	"num_tokens": 1321412178.0,
	"step": 319
	},
	{
	"epoch": 1.131162069347513,
	"grad_norm": 0.16598292517459642,
	"learning_rate": 1.91790929996278e-05,
	"loss": 0.1308,
	"num_tokens": 1325558749.0,
	"step": 320
	},
	{
	"epoch": 1.134706990140689,
	"grad_norm": 0.15545476660292895,
	"learning_rate": 1.9077409456046258e-05,
	"loss": 0.1189,
	"num_tokens": 1329714175.0,
	"step": 321
	},
	{
	"epoch": 1.1382519109338651,
	"grad_norm": 0.13551213670950768,
	"learning_rate": 1.8975821614648123e-05,
	"loss": 0.1229,
	"num_tokens": 1333874359.0,
	"step": 322
	},
	{
	"epoch": 1.141796831727041,
	"grad_norm": 0.16854458763059854,
	"learning_rate": 1.8874332801995258e-05,
	"loss": 0.1189,
	"num_tokens": 1338008560.0,
	"step": 323
	},
	{
	"epoch": 1.1453417525202172,
	"grad_norm": 0.18459750330652283,
	"learning_rate": 1.8772946341406767e-05,
	"loss": 0.127,
	"num_tokens": 1342167052.0,
	"step": 324
	},
	{
	"epoch": 1.1488866733133931,
	"grad_norm": 0.14864065305084023,
	"learning_rate": 1.867166555285014e-05,
	"loss": 0.1279,
	"num_tokens": 1346336963.0,
	"step": 325
	},
	{
	"epoch": 1.1524315941065693,
	"grad_norm": 0.12194569945074321,
	"learning_rate": 1.8570493752832603e-05,
	"loss": 0.1213,
	"num_tokens": 1350495965.0,
	"step": 326
	},
	{
	"epoch": 1.1559765148997452,
	"grad_norm": 0.1511241690115024,
	"learning_rate": 1.8469434254292456e-05,
	"loss": 0.1273,
	"num_tokens": 1354622402.0,
	"step": 327
	},
	{
	"epoch": 1.1595214356929213,
	"grad_norm": 0.16535966245136396,
	"learning_rate": 1.8368490366490612e-05,
	"loss": 0.1257,
	"num_tokens": 1358769481.0,
	"step": 328
	},
	{
	"epoch": 1.1630663564860972,
	"grad_norm": 0.13790597542822972,
	"learning_rate": 1.826766539490224e-05,
	"loss": 0.1243,
	"num_tokens": 1362919302.0,
	"step": 329
	},
	{
	"epoch": 1.1666112772792734,
	"grad_norm": 0.13308498711452516,
	"learning_rate": 1.8166962641108525e-05,
	"loss": 0.1263,
	"num_tokens": 1367084296.0,
	"step": 330
	},
	{
	"epoch": 1.1701561980724493,
	"grad_norm": 0.15699129290069,
	"learning_rate": 1.8066385402688516e-05,
	"loss": 0.1217,
	"num_tokens": 1371236399.0,
	"step": 331
	},
	{
	"epoch": 1.1737011188656252,
	"grad_norm": 0.15367646160249687,
	"learning_rate": 1.7965936973111215e-05,
	"loss": 0.1271,
	"num_tokens": 1375420795.0,
	"step": 332
	},
	{
	"epoch": 1.1772460396588014,
	"grad_norm": 0.14614684379868828,
	"learning_rate": 1.7865620641627686e-05,
	"loss": 0.1239,
	"num_tokens": 1379608169.0,
	"step": 333
	},
	{
	"epoch": 1.1807909604519775,
	"grad_norm": 0.12166199774820369,
	"learning_rate": 1.7765439693163325e-05,
	"loss": 0.1273,
	"num_tokens": 1383743315.0,
	"step": 334
	},
	{
	"epoch": 1.1843358812451534,
	"grad_norm": 0.13090367483861337,
	"learning_rate": 1.7665397408210348e-05,
	"loss": 0.1248,
	"num_tokens": 1387898613.0,
	"step": 335
	},
	{
	"epoch": 1.1878808020383294,
	"grad_norm": 0.15143502282522925,
	"learning_rate": 1.7565497062720335e-05,
	"loss": 0.1274,
	"num_tokens": 1392028168.0,
	"step": 336
	},
	{
	"epoch": 1.1914257228315055,
	"grad_norm": 0.13959726997564648,
	"learning_rate": 1.746574192799696e-05,
	"loss": 0.1253,
	"num_tokens": 1396176552.0,
	"step": 337
	},
	{
	"epoch": 1.1949706436246814,
	"grad_norm": 0.12810494232099812,
	"learning_rate": 1.7366135270588875e-05,
	"loss": 0.1254,
	"num_tokens": 1400304582.0,
	"step": 338
	},
	{
	"epoch": 1.1985155644178576,
	"grad_norm": 0.13068815380503193,
	"learning_rate": 1.7266680352182737e-05,
	"loss": 0.1226,
	"num_tokens": 1404458875.0,
	"step": 339
	},
	{
	"epoch": 1.2020604852110335,
	"grad_norm": 0.12937935587005236,
	"learning_rate": 1.7167380429496417e-05,
	"loss": 0.1213,
	"num_tokens": 1408641787.0,
	"step": 340
	},
	{
	"epoch": 1.2056054060042096,
	"grad_norm": 0.11388926877778172,
	"learning_rate": 1.7068238754172347e-05,
	"loss": 0.1181,
	"num_tokens": 1412770204.0,
	"step": 341
	},
	{
	"epoch": 1.2091503267973855,
	"grad_norm": 0.11670716564963377,
	"learning_rate": 1.6969258572671036e-05,
	"loss": 0.1254,
	"num_tokens": 1416915010.0,
	"step": 342
	},
	{
	"epoch": 1.2126952475905617,
	"grad_norm": 0.1123504509403795,
	"learning_rate": 1.6870443126164766e-05,
	"loss": 0.1266,
	"num_tokens": 1421069510.0,
	"step": 343
	},
	{
	"epoch": 1.2162401683837376,
	"grad_norm": 0.11928538781368539,
	"learning_rate": 1.6771795650431485e-05,
	"loss": 0.1265,
	"num_tokens": 1425229346.0,
	"step": 344
	},
	{
	"epoch": 1.2197850891769138,
	"grad_norm": 0.12291911866225681,
	"learning_rate": 1.6673319375748796e-05,
	"loss": 0.1204,
	"num_tokens": 1429411928.0,
	"step": 345
	},
	{
	"epoch": 1.2233300099700897,
	"grad_norm": 0.1325769988967944,
	"learning_rate": 1.6575017526788224e-05,
	"loss": 0.1204,
	"num_tokens": 1433562610.0,
	"step": 346
	},
	{
	"epoch": 1.2268749307632658,
	"grad_norm": 0.1144642040678278,
	"learning_rate": 1.6476893322509613e-05,
	"loss": 0.1248,
	"num_tokens": 1437729476.0,
	"step": 347
	},
	{
	"epoch": 1.2304198515564417,
	"grad_norm": 0.12901758406986696,
	"learning_rate": 1.63789499760557e-05,
	"loss": 0.1234,
	"num_tokens": 1441892065.0,
	"step": 348
	},
	{
	"epoch": 1.2339647723496179,
	"grad_norm": 0.1264730200871411,
	"learning_rate": 1.6281190694646907e-05,
	"loss": 0.1205,
	"num_tokens": 1446075085.0,
	"step": 349
	},
	{
	"epoch": 1.2375096931427938,
	"grad_norm": 0.11992500472389374,
	"learning_rate": 1.6183618679476364e-05,
	"loss": 0.123,
	"num_tokens": 1450202168.0,
	"step": 350
	},
	{
	"epoch": 1.24105461393597,
	"grad_norm": 0.13560844342683573,
	"learning_rate": 1.6086237125604974e-05,
	"loss": 0.1248,
	"num_tokens": 1454331778.0,
	"step": 351
	},
	{
	"epoch": 1.2445995347291459,
	"grad_norm": 0.11260924987662181,
	"learning_rate": 1.598904922185692e-05,
	"loss": 0.1205,
	"num_tokens": 1458502818.0,
	"step": 352
	},
	{
	"epoch": 1.248144455522322,
	"grad_norm": 0.14098351197742362,
	"learning_rate": 1.5892058150715146e-05,
	"loss": 0.1201,
	"num_tokens": 1462647250.0,
	"step": 353
	},
	{
	"epoch": 1.251689376315498,
	"grad_norm": 0.1402415903309477,
	"learning_rate": 1.579526708821719e-05,
	"loss": 0.1232,
	"num_tokens": 1466784306.0,
	"step": 354
	},
	{
	"epoch": 1.2552342971086738,
	"grad_norm": 0.09632733727583728,
	"learning_rate": 1.5698679203851183e-05,
	"loss": 0.1235,
	"num_tokens": 1470930200.0,
	"step": 355
	},
	{
	"epoch": 1.25877921790185,
	"grad_norm": 0.14782331208379643,
	"learning_rate": 1.560229766045204e-05,
	"loss": 0.123,
	"num_tokens": 1475078612.0,
	"step": 356
	},
	{
	"epoch": 1.2623241386950261,
	"grad_norm": 0.1616324987785373,
	"learning_rate": 1.5506125614097903e-05,
	"loss": 0.1218,
	"num_tokens": 1479229388.0,
	"step": 357
	},
	{
	"epoch": 1.265869059488202,
	"grad_norm": 0.12324762797406584,
	"learning_rate": 1.54101662140068e-05,
	"loss": 0.1186,
	"num_tokens": 1483374024.0,
	"step": 358
	},
	{
	"epoch": 1.269413980281378,
	"grad_norm": 0.12250148194150823,
	"learning_rate": 1.531442260243351e-05,
	"loss": 0.1235,
	"num_tokens": 1487554314.0,
	"step": 359
	},
	{
	"epoch": 1.2729589010745541,
	"grad_norm": 0.1577091584861214,
	"learning_rate": 1.5218897914566668e-05,
	"loss": 0.1175,
	"num_tokens": 1491724303.0,
	"step": 360
	},
	{
	"epoch": 1.2765038218677303,
	"grad_norm": 0.13417797107822863,
	"learning_rate": 1.5123595278426133e-05,
	"loss": 0.1222,
	"num_tokens": 1495908099.0,
	"step": 361
	},
	{
	"epoch": 1.2800487426609062,
	"grad_norm": 0.10688316410411597,
	"learning_rate": 1.5028517814760482e-05,
	"loss": 0.1234,
	"num_tokens": 1500068497.0,
	"step": 362
	},
	{
	"epoch": 1.283593663454082,
	"grad_norm": 0.13723257379804255,
	"learning_rate": 1.4933668636944912e-05,
	"loss": 0.121,
	"num_tokens": 1504234360.0,
	"step": 363
	},
	{
	"epoch": 1.2871385842472582,
	"grad_norm": 0.10581373304115722,
	"learning_rate": 1.483905085087923e-05,
	"loss": 0.1191,
	"num_tokens": 1508388685.0,
	"step": 364
	},
	{
	"epoch": 1.2906835050404344,
	"grad_norm": 0.13427149149019874,
	"learning_rate": 1.4744667554886167e-05,
	"loss": 0.1238,
	"num_tokens": 1512566570.0,
	"step": 365
	},
	{
	"epoch": 1.2942284258336103,
	"grad_norm": 0.11281114861275712,
	"learning_rate": 1.4650521839609919e-05,
	"loss": 0.1181,
	"num_tokens": 1516747493.0,
	"step": 366
	},
	{
	"epoch": 1.2977733466267862,
	"grad_norm": 0.10504434030797455,
	"learning_rate": 1.4556616787914948e-05,
	"loss": 0.1249,
	"num_tokens": 1520896897.0,
	"step": 367
	},
	{
	"epoch": 1.3013182674199624,
	"grad_norm": 0.11129478599659645,
	"learning_rate": 1.4462955474785014e-05,
	"loss": 0.122,
	"num_tokens": 1525060483.0,
	"step": 368
	},
	{
	"epoch": 1.3048631882131383,
	"grad_norm": 0.11735525944127165,
	"learning_rate": 1.4369540967222512e-05,
	"loss": 0.1168,
	"num_tokens": 1529213811.0,
	"step": 369
	},
	{
	"epoch": 1.3084081090063144,
	"grad_norm": 0.09516546971310873,
	"learning_rate": 1.4276376324148005e-05,
	"loss": 0.1211,
	"num_tokens": 1533380123.0,
	"step": 370
	},
	{
	"epoch": 1.3119530297994904,
	"grad_norm": 0.09692048458576819,
	"learning_rate": 1.4183464596300076e-05,
	"loss": 0.1236,
	"num_tokens": 1537537330.0,
	"step": 371
	},
	{
	"epoch": 1.3154979505926665,
	"grad_norm": 0.10453881003578407,
	"learning_rate": 1.409080882613545e-05,
	"loss": 0.1238,
	"num_tokens": 1541714257.0,
	"step": 372
	},
	{
	"epoch": 1.3190428713858424,
	"grad_norm": 0.08783083401479898,
	"learning_rate": 1.3998412047729337e-05,
	"loss": 0.1207,
	"num_tokens": 1545883894.0,
	"step": 373
	},
	{
	"epoch": 1.3225877921790186,
	"grad_norm": 0.11809181808369248,
	"learning_rate": 1.3906277286676085e-05,
	"loss": 0.1207,
	"num_tokens": 1550034009.0,
	"step": 374
	},
	{
	"epoch": 1.3261327129721945,
	"grad_norm": 0.11613371385814988,
	"learning_rate": 1.3814407559990102e-05,
	"loss": 0.1206,
	"num_tokens": 1554183351.0,
	"step": 375
	},
	{
	"epoch": 1.3296776337653706,
	"grad_norm": 0.10864401527961921,
	"learning_rate": 1.3722805876007073e-05,
	"loss": 0.115,
	"num_tokens": 1558343191.0,
	"step": 376
	},
	{
	"epoch": 1.3332225545585465,
	"grad_norm": 0.1028456113005129,
	"learning_rate": 1.3631475234285456e-05,
	"loss": 0.1264,
	"num_tokens": 1562439299.0,
	"step": 377
	},
	{
	"epoch": 1.3367674753517227,
	"grad_norm": 0.1215954095819048,
	"learning_rate": 1.3540418625508247e-05,
	"loss": 0.1177,
	"num_tokens": 1566619792.0,
	"step": 378
	},
	{
	"epoch": 1.3403123961448986,
	"grad_norm": 0.10806349361544453,
	"learning_rate": 1.3449639031385026e-05,
	"loss": 0.1198,
	"num_tokens": 1570768441.0,
	"step": 379
	},
	{
	"epoch": 1.3438573169380748,
	"grad_norm": 0.1029573674311725,
	"learning_rate": 1.3359139424554366e-05,
	"loss": 0.1185,
	"num_tokens": 1574909645.0,
	"step": 380
	},
	{
	"epoch": 1.3474022377312507,
	"grad_norm": 0.09968451301036962,
	"learning_rate": 1.3268922768486472e-05,
	"loss": 0.1186,
	"num_tokens": 1579058664.0,
	"step": 381
	},
	{
	"epoch": 1.3509471585244266,
	"grad_norm": 0.11172568204860428,
	"learning_rate": 1.3178992017386122e-05,
	"loss": 0.1218,
	"num_tokens": 1583222046.0,
	"step": 382
	},
	{
	"epoch": 1.3544920793176027,
	"grad_norm": 0.11963230763471129,
	"learning_rate": 1.3089350116095945e-05,
	"loss": 0.1265,
	"num_tokens": 1587366257.0,
	"step": 383
	},
	{
	"epoch": 1.3580370001107789,
	"grad_norm": 0.10646682160466439,
	"learning_rate": 1.3000000000000006e-05,
	"loss": 0.1224,
	"num_tokens": 1591526074.0,
	"step": 384
	},
	{
	"epoch": 1.3615819209039548,
	"grad_norm": 0.09961412811586683,
	"learning_rate": 1.2910944594927642e-05,
	"loss": 0.1196,
	"num_tokens": 1595684855.0,
	"step": 385
	},
	{
	"epoch": 1.3651268416971307,
	"grad_norm": 0.1343730427241239,
	"learning_rate": 1.2822186817057714e-05,
	"loss": 0.1228,
	"num_tokens": 1599859718.0,
	"step": 386
	},
	{
	"epoch": 1.3686717624903069,
	"grad_norm": 0.10298167804077214,
	"learning_rate": 1.2733729572823053e-05,
	"loss": 0.1221,
	"num_tokens": 1604011888.0,
	"step": 387
	},
	{
	"epoch": 1.372216683283483,
	"grad_norm": 0.11149906504704246,
	"learning_rate": 1.2645575758815321e-05,
	"loss": 0.117,
	"num_tokens": 1608175425.0,
	"step": 388
	},
	{
	"epoch": 1.375761604076659,
	"grad_norm": 0.12556178260153125,
	"learning_rate": 1.2557728261690158e-05,
	"loss": 0.1219,
	"num_tokens": 1612318661.0,
	"step": 389
	},
	{
	"epoch": 1.3793065248698348,
	"grad_norm": 0.09756718152235203,
	"learning_rate": 1.2470189958072668e-05,
	"loss": 0.1234,
	"num_tokens": 1616490111.0,
	"step": 390
	},
	{
	"epoch": 1.382851445663011,
	"grad_norm": 0.11431698214260155,
	"learning_rate": 1.2382963714463177e-05,
	"loss": 0.121,
	"num_tokens": 1620666519.0,
	"step": 391
	},
	{
	"epoch": 1.386396366456187,
	"grad_norm": 0.10228330727622398,
	"learning_rate": 1.229605238714341e-05,
	"loss": 0.1225,
	"num_tokens": 1624813106.0,
	"step": 392
	},
	{
	"epoch": 1.389941287249363,
	"grad_norm": 0.09895107753249643,
	"learning_rate": 1.2209458822082957e-05,
	"loss": 0.1176,
	"num_tokens": 1628972537.0,
	"step": 393
	},
	{
	"epoch": 1.393486208042539,
	"grad_norm": 0.11596615708480393,
	"learning_rate": 1.2123185854846055e-05,
	"loss": 0.1162,
	"num_tokens": 1633133820.0,
	"step": 394
	},
	{
	"epoch": 1.3970311288357151,
	"grad_norm": 0.10150974881101132,
	"learning_rate": 1.2037236310498766e-05,
	"loss": 0.1265,
	"num_tokens": 1637299270.0,
	"step": 395
	},
	{
	"epoch": 1.400576049628891,
	"grad_norm": 0.10014580534642231,
	"learning_rate": 1.1951613003516438e-05,
	"loss": 0.1165,
	"num_tokens": 1641440658.0,
	"step": 396
	},
	{
	"epoch": 1.4041209704220672,
	"grad_norm": 0.11107178667041182,
	"learning_rate": 1.1866318737691562e-05,
	"loss": 0.1175,
	"num_tokens": 1645613495.0,
	"step": 397
	},
	{
	"epoch": 1.407665891215243,
	"grad_norm": 0.09291127575893779,
	"learning_rate": 1.1781356306041967e-05,
	"loss": 0.1204,
	"num_tokens": 1649787740.0,
	"step": 398
	},
	{
	"epoch": 1.4112108120084192,
	"grad_norm": 0.1071850002682706,
	"learning_rate": 1.1696728490719335e-05,
	"loss": 0.1139,
	"num_tokens": 1653956617.0,
	"step": 399
	},
	{
	"epoch": 1.4147557328015952,
	"grad_norm": 0.11688932972665554,
	"learning_rate": 1.1612438062918124e-05,
	"loss": 0.123,
	"num_tokens": 1658084550.0,
	"step": 400
	},
	{
	"epoch": 1.4183006535947713,
	"grad_norm": 0.10363546445362747,
	"learning_rate": 1.1528487782784805e-05,
	"loss": 0.1189,
	"num_tokens": 1662258894.0,
	"step": 401
	},
	{
	"epoch": 1.4218455743879472,
	"grad_norm": 0.11547961526749512,
	"learning_rate": 1.144488039932748e-05,
	"loss": 0.1189,
	"num_tokens": 1666428334.0,
	"step": 402
	},
	{
	"epoch": 1.4253904951811234,
	"grad_norm": 0.11093380814757749,
	"learning_rate": 1.1361618650325885e-05,
	"loss": 0.123,
	"num_tokens": 1670588116.0,
	"step": 403
	},
	{
	"epoch": 1.4289354159742993,
	"grad_norm": 0.08954670694686448,
	"learning_rate": 1.1278705262241725e-05,
	"loss": 0.1213,
	"num_tokens": 1674774095.0,
	"step": 404
	},
	{
	"epoch": 1.4324803367674752,
	"grad_norm": 0.09901851364835385,
	"learning_rate": 1.1196142950129375e-05,
	"loss": 0.1194,
	"num_tokens": 1678928700.0,
	"step": 405
	},
	{
	"epoch": 1.4360252575606514,
	"grad_norm": 0.09374065635086802,
	"learning_rate": 1.1113934417547e-05,
	"loss": 0.1172,
	"num_tokens": 1683093022.0,
	"step": 406
	},
	{
	"epoch": 1.4395701783538275,
	"grad_norm": 0.09051074227184891,
	"learning_rate": 1.1032082356468035e-05,
	"loss": 0.122,
	"num_tokens": 1687248699.0,
	"step": 407
	},
	{
	"epoch": 1.4431150991470034,
	"grad_norm": 0.09634393456760387,
	"learning_rate": 1.0950589447192982e-05,
	"loss": 0.1202,
	"num_tokens": 1691386638.0,
	"step": 408
	},
	{
	"epoch": 1.4466600199401793,
	"grad_norm": 0.09599812757277522,
	"learning_rate": 1.086945835826172e-05,
	"loss": 0.1213,
	"num_tokens": 1695515101.0,
	"step": 409
	},
	{
	"epoch": 1.4502049407333555,
	"grad_norm": 0.09710849082824417,
	"learning_rate": 1.0788691746366045e-05,
	"loss": 0.1173,
	"num_tokens": 1699668023.0,
	"step": 410
	},
	{
	"epoch": 1.4537498615265316,
	"grad_norm": 0.10108972589998338,
	"learning_rate": 1.0708292256262714e-05,
	"loss": 0.1226,
	"num_tokens": 1703822556.0,
	"step": 411
	},
	{
	"epoch": 1.4572947823197075,
	"grad_norm": 0.10915137689149003,
	"learning_rate": 1.0628262520686849e-05,
	"loss": 0.1238,
	"num_tokens": 1708001633.0,
	"step": 412
	},
	{
	"epoch": 1.4608397031128835,
	"grad_norm": 0.09178625174550167,
	"learning_rate": 1.0548605160265697e-05,
	"loss": 0.121,
	"num_tokens": 1712169388.0,
	"step": 413
	},
	{
	"epoch": 1.4643846239060596,
	"grad_norm": 0.09269505309054628,
	"learning_rate": 1.046932278343285e-05,
	"loss": 0.1185,
	"num_tokens": 1716311394.0,
	"step": 414
	},
	{
	"epoch": 1.4679295446992358,
	"grad_norm": 0.09188721133030069,
	"learning_rate": 1.0390417986342794e-05,
	"loss": 0.1213,
	"num_tokens": 1720496242.0,
	"step": 415
	},
	{
	"epoch": 1.4714744654924117,
	"grad_norm": 0.09455830746786394,
	"learning_rate": 1.0311893352785912e-05,
	"loss": 0.1189,
	"num_tokens": 1724638950.0,
	"step": 416
	},
	{
	"epoch": 1.4750193862855876,
	"grad_norm": 0.08996759077523822,
	"learning_rate": 1.0233751454103893e-05,
	"loss": 0.1185,
	"num_tokens": 1728811816.0,
	"step": 417
	},
	{
	"epoch": 1.4785643070787637,
	"grad_norm": 0.09137684567332999,
	"learning_rate": 1.0155994849105518e-05,
	"loss": 0.1166,
	"num_tokens": 1732946294.0,
	"step": 418
	},
	{
	"epoch": 1.4821092278719397,
	"grad_norm": 0.09183690083171721,
	"learning_rate": 1.0078626083982839e-05,
	"loss": 0.1237,
	"num_tokens": 1737105101.0,
	"step": 419
	},
	{
	"epoch": 1.4856541486651158,
	"grad_norm": 0.08679668069909362,
	"learning_rate": 1.0001647692227862e-05,
	"loss": 0.1224,
	"num_tokens": 1741279010.0,
	"step": 420
	},
	{
	"epoch": 1.4891990694582917,
	"grad_norm": 0.08658932913907078,
	"learning_rate": 9.925062194549553e-06,
	"loss": 0.1212,
	"num_tokens": 1745427370.0,
	"step": 421
	},
	{
	"epoch": 1.4927439902514679,
	"grad_norm": 0.0858620951997791,
	"learning_rate": 9.848872098791284e-06,
	"loss": 0.1201,
	"num_tokens": 1749602356.0,
	"step": 422
	},
	{
	"epoch": 1.4962889110446438,
	"grad_norm": 0.0854929448511193,
	"learning_rate": 9.773079899848725e-06,
	"loss": 0.1186,
	"num_tokens": 1753744559.0,
	"step": 423
	},
	{
	"epoch": 1.49983383183782,
	"grad_norm": 0.09235833823210157,
	"learning_rate": 9.697688079588174e-06,
	"loss": 0.1224,
	"num_tokens": 1757896457.0,
	"step": 424
	},
	{
	"epoch": 1.5033787526309959,
	"grad_norm": 0.08772354377779477,
	"learning_rate": 9.62269910676523e-06,
	"loss": 0.121,
	"num_tokens": 1761993831.0,
	"step": 425
	},
	{
	"epoch": 1.506923673424172,
	"grad_norm": 0.09554255775608568,
	"learning_rate": 9.548115436944008e-06,
	"loss": 0.1188,
	"num_tokens": 1766152380.0,
	"step": 426
	},
	{
	"epoch": 1.510468594217348,
	"grad_norm": 0.08118180697902405,
	"learning_rate": 9.47393951241669e-06,
	"loss": 0.1198,
	"num_tokens": 1770320990.0,
	"step": 427
	},
	{
	"epoch": 1.5140135150105238,
	"grad_norm": 0.09565462445407204,
	"learning_rate": 9.400173762123564e-06,
	"loss": 0.1179,
	"num_tokens": 1774479657.0,
	"step": 428
	},
	{
	"epoch": 1.5175584358037,
	"grad_norm": 0.08391365839157391,
	"learning_rate": 9.326820601573497e-06,
	"loss": 0.1247,
	"num_tokens": 1778626444.0,
	"step": 429
	},
	{
	"epoch": 1.5211033565968761,
	"grad_norm": 0.0809083083208967,
	"learning_rate": 9.253882432764835e-06,
	"loss": 0.113,
	"num_tokens": 1782805208.0,
	"step": 430
	},
	{
	"epoch": 1.524648277390052,
	"grad_norm": 0.08965375342019177,
	"learning_rate": 9.181361644106723e-06,
	"loss": 0.1152,
	"num_tokens": 1786915825.0,
	"step": 431
	},
	{
	"epoch": 1.528193198183228,
	"grad_norm": 0.08896279165359179,
	"learning_rate": 9.109260610340925e-06,
	"loss": 0.1201,
	"num_tokens": 1791059824.0,
	"step": 432
	},
	{
	"epoch": 1.531738118976404,
	"grad_norm": 0.08911156097429918,
	"learning_rate": 9.037581692464042e-06,
	"loss": 0.1168,
	"num_tokens": 1795237132.0,
	"step": 433
	},
	{
	"epoch": 1.5352830397695802,
	"grad_norm": 0.09745379832957693,
	"learning_rate": 8.966327237650221e-06,
	"loss": 0.1143,
	"num_tokens": 1799393689.0,
	"step": 434
	},
	{
	"epoch": 1.5388279605627562,
	"grad_norm": 0.0847162122003079,
	"learning_rate": 8.895499579174275e-06,
	"loss": 0.1212,
	"num_tokens": 1803543885.0,
	"step": 435
	},
	{
	"epoch": 1.542372881355932,
	"grad_norm": 0.09452318364660556,
	"learning_rate": 8.825101036335287e-06,
	"loss": 0.1144,
	"num_tokens": 1807693998.0,
	"step": 436
	},
	{
	"epoch": 1.5459178021491082,
	"grad_norm": 0.08440688296040805,
	"learning_rate": 8.755133914380644e-06,
	"loss": 0.1173,
	"num_tokens": 1811869581.0,
	"step": 437
	},
	{
	"epoch": 1.5494627229422844,
	"grad_norm": 0.09839744574886089,
	"learning_rate": 8.6856005044306e-06,
	"loss": 0.123,
	"num_tokens": 1816035224.0,
	"step": 438
	},
	{
	"epoch": 1.5530076437354603,
	"grad_norm": 0.09323185793448917,
	"learning_rate": 8.616503083403192e-06,
	"loss": 0.1162,
	"num_tokens": 1820169185.0,
	"step": 439
	},
	{
	"epoch": 1.5565525645286362,
	"grad_norm": 0.08573460091310847,
	"learning_rate": 8.547843913939723e-06,
	"loss": 0.1181,
	"num_tokens": 1824338389.0,
	"step": 440
	},
	{
	"epoch": 1.5600974853218124,
	"grad_norm": 0.09094729672604082,
	"learning_rate": 8.479625244330648e-06,
	"loss": 0.1196,
	"num_tokens": 1828482652.0,
	"step": 441
	},
	{
	"epoch": 1.5636424061149885,
	"grad_norm": 0.083096796985677,
	"learning_rate": 8.411849308441956e-06,
	"loss": 0.1202,
	"num_tokens": 1832651754.0,
	"step": 442
	},
	{
	"epoch": 1.5671873269081644,
	"grad_norm": 0.084193678248527,
	"learning_rate": 8.344518325642029e-06,
	"loss": 0.1185,
	"num_tokens": 1836767991.0,
	"step": 443
	},
	{
	"epoch": 1.5707322477013403,
	"grad_norm": 0.08686366703241588,
	"learning_rate": 8.277634500728976e-06,
	"loss": 0.1187,
	"num_tokens": 1840920891.0,
	"step": 444
	},
	{
	"epoch": 1.5742771684945165,
	"grad_norm": 0.08181203717834039,
	"learning_rate": 8.2112000238584e-06,
	"loss": 0.1198,
	"num_tokens": 1845071226.0,
	"step": 445
	},
	{
	"epoch": 1.5778220892876926,
	"grad_norm": 0.08590293115074739,
	"learning_rate": 8.1452170704717e-06,
	"loss": 0.1217,
	"num_tokens": 1849230810.0,
	"step": 446
	},
	{
	"epoch": 1.5813670100808686,
	"grad_norm": 0.08838846412588158,
	"learning_rate": 8.079687801224864e-06,
	"loss": 0.119,
	"num_tokens": 1853395093.0,
	"step": 447
	},
	{
	"epoch": 1.5849119308740445,
	"grad_norm": 0.07825846635223611,
	"learning_rate": 8.014614361917657e-06,
	"loss": 0.1209,
	"num_tokens": 1857552268.0,
	"step": 448
	},
	{
	"epoch": 1.5884568516672206,
	"grad_norm": 0.07962888111838136,
	"learning_rate": 7.949998883423414e-06,
	"loss": 0.1193,
	"num_tokens": 1861676962.0,
	"step": 449
	},
	{
	"epoch": 1.5920017724603965,
	"grad_norm": 0.0769694677854603,
	"learning_rate": 7.885843481619215e-06,
	"loss": 0.1129,
	"num_tokens": 1865844162.0,
	"step": 450
	},
	{
	"epoch": 1.5955466932535725,
	"grad_norm": 0.08582651457499006,
	"learning_rate": 7.822150257316632e-06,
	"loss": 0.1215,
	"num_tokens": 1869998942.0,
	"step": 451
	},
	{
	"epoch": 1.5990916140467486,
	"grad_norm": 0.08369837488083547,
	"learning_rate": 7.75892129619293e-06,
	"loss": 0.1181,
	"num_tokens": 1874139183.0,
	"step": 452
	},
	{
	"epoch": 1.6026365348399247,
	"grad_norm": 0.08943248840302526,
	"learning_rate": 7.696158668722748e-06,
	"loss": 0.1185,
	"num_tokens": 1878298627.0,
	"step": 453
	},
	{
	"epoch": 1.6061814556331007,
	"grad_norm": 0.08549962402484389,
	"learning_rate": 7.633864430110342e-06,
	"loss": 0.1192,
	"num_tokens": 1882437488.0,
	"step": 454
	},
	{
	"epoch": 1.6097263764262766,
	"grad_norm": 0.09596343326710725,
	"learning_rate": 7.572040620222243e-06,
	"loss": 0.1208,
	"num_tokens": 1886573720.0,
	"step": 455
	},
	{
	"epoch": 1.6132712972194527,
	"grad_norm": 0.08110606373110157,
	"learning_rate": 7.510689263520475e-06,
	"loss": 0.1175,
	"num_tokens": 1890721290.0,
	"step": 456
	},
	{
	"epoch": 1.6168162180126289,
	"grad_norm": 0.08190211879874516,
	"learning_rate": 7.449812368996283e-06,
	"loss": 0.1213,
	"num_tokens": 1894871055.0,
	"step": 457
	},
	{
	"epoch": 1.6203611388058048,
	"grad_norm": 0.08688397528875834,
	"learning_rate": 7.389411930104331e-06,
	"loss": 0.1185,
	"num_tokens": 1899034118.0,
	"step": 458
	},
	{
	"epoch": 1.6239060595989807,
	"grad_norm": 0.09415089014287763,
	"learning_rate": 7.329489924697398e-06,
	"loss": 0.1181,
	"num_tokens": 1903172800.0,
	"step": 459
	},
	{
	"epoch": 1.6274509803921569,
	"grad_norm": 0.08280572540971591,
	"learning_rate": 7.270048314961664e-06,
	"loss": 0.1215,
	"num_tokens": 1907334246.0,
	"step": 460
	},
	{
	"epoch": 1.630995901185333,
	"grad_norm": 0.09265143470986054,
	"learning_rate": 7.2110890473524284e-06,
	"loss": 0.1171,
	"num_tokens": 1911469127.0,
	"step": 461
	},
	{
	"epoch": 1.634540821978509,
	"grad_norm": 0.08194132476505427,
	"learning_rate": 7.15261405253036e-06,
	"loss": 0.1205,
	"num_tokens": 1915635345.0,
	"step": 462
	},
	{
	"epoch": 1.6380857427716848,
	"grad_norm": 0.08361869458895219,
	"learning_rate": 7.094625245298312e-06,
	"loss": 0.1189,
	"num_tokens": 1919764869.0,
	"step": 463
	},
	{
	"epoch": 1.641630663564861,
	"grad_norm": 0.0845459420140274,
	"learning_rate": 7.037124524538581e-06,
	"loss": 0.1197,
	"num_tokens": 1923939327.0,
	"step": 464
	},
	{
	"epoch": 1.6451755843580371,
	"grad_norm": 0.07714422760112216,
	"learning_rate": 6.980113773150749e-06,
	"loss": 0.1163,
	"num_tokens": 1928087767.0,
	"step": 465
	},
	{
	"epoch": 1.648720505151213,
	"grad_norm": 0.08716574358691197,
	"learning_rate": 6.923594857990034e-06,
	"loss": 0.1191,
	"num_tokens": 1932245216.0,
	"step": 466
	},
	{
	"epoch": 1.652265425944389,
	"grad_norm": 0.07921380817575643,
	"learning_rate": 6.867569629806145e-06,
	"loss": 0.1106,
	"num_tokens": 1936431022.0,
	"step": 467
	},
	{
	"epoch": 1.655810346737565,
	"grad_norm": 0.09467616408530952,
	"learning_rate": 6.8120399231826595e-06,
	"loss": 0.1259,
	"num_tokens": 1940579685.0,
	"step": 468
	},
	{
	"epoch": 1.6593552675307413,
	"grad_norm": 0.08042506658742254,
	"learning_rate": 6.757007556477001e-06,
	"loss": 0.113,
	"num_tokens": 1944747366.0,
	"step": 469
	},
	{
	"epoch": 1.6629001883239172,
	"grad_norm": 0.0930859404297044,
	"learning_rate": 6.702474331760842e-06,
	"loss": 0.1153,
	"num_tokens": 1948900361.0,
	"step": 470
	},
	{
	"epoch": 1.666445109117093,
	"grad_norm": 0.07425838242566188,
	"learning_rate": 6.648442034761129e-06,
	"loss": 0.1139,
	"num_tokens": 1953066926.0,
	"step": 471
	},
	{
	"epoch": 1.6699900299102692,
	"grad_norm": 0.0906296892316949,
	"learning_rate": 6.5949124348015856e-06,
	"loss": 0.1144,
	"num_tokens": 1957243243.0,
	"step": 472
	},
	{
	"epoch": 1.6735349507034454,
	"grad_norm": 0.07558541347255775,
	"learning_rate": 6.541887284744792e-06,
	"loss": 0.1208,
	"num_tokens": 1961380330.0,
	"step": 473
	},
	{
	"epoch": 1.6770798714966213,
	"grad_norm": 0.0812226966971371,
	"learning_rate": 6.489368320934773e-06,
	"loss": 0.1139,
	"num_tokens": 1965527463.0,
	"step": 474
	},
	{
	"epoch": 1.6806247922897972,
	"grad_norm": 0.08140590796517343,
	"learning_rate": 6.437357263140156e-06,
	"loss": 0.1139,
	"num_tokens": 1969646132.0,
	"step": 475
	},
	{
	"epoch": 1.6841697130829734,
	"grad_norm": 0.08068228101222655,
	"learning_rate": 6.385855814497832e-06,
	"loss": 0.1187,
	"num_tokens": 1973825626.0,
	"step": 476
	},
	{
	"epoch": 1.6877146338761493,
	"grad_norm": 0.0698026034904853,
	"learning_rate": 6.3348656614572055e-06,
	"loss": 0.1125,
	"num_tokens": 1977995058.0,
	"step": 477
	},
	{
	"epoch": 1.6912595546693252,
	"grad_norm": 0.08183260030682185,
	"learning_rate": 6.284388473724974e-06,
	"loss": 0.1182,
	"num_tokens": 1982136461.0,
	"step": 478
	},
	{
	"epoch": 1.6948044754625013,
	"grad_norm": 0.08630474997023979,
	"learning_rate": 6.234425904210419e-06,
	"loss": 0.1166,
	"num_tokens": 1986309800.0,
	"step": 479
	},
	{
	"epoch": 1.6983493962556775,
	"grad_norm": 0.07685498117174024,
	"learning_rate": 6.1849795889713335e-06,
	"loss": 0.1189,
	"num_tokens": 1990454655.0,
	"step": 480
	},
	{
	"epoch": 1.7018943170488534,
	"grad_norm": 0.07873020745183176,
	"learning_rate": 6.1360511471603945e-06,
	"loss": 0.1153,
	"num_tokens": 1994604908.0,
	"step": 481
	},
	{
	"epoch": 1.7054392378420293,
	"grad_norm": 0.07574585808955839,
	"learning_rate": 6.087642180972178e-06,
	"loss": 0.1159,
	"num_tokens": 1998750349.0,
	"step": 482
	},
	{
	"epoch": 1.7089841586352055,
	"grad_norm": 0.08201734773812737,
	"learning_rate": 6.039754275590678e-06,
	"loss": 0.1185,
	"num_tokens": 2002896465.0,
	"step": 483
	},
	{
	"epoch": 1.7125290794283816,
	"grad_norm": 0.07639926482051736,
	"learning_rate": 5.9923889991374176e-06,
	"loss": 0.1231,
	"num_tokens": 2007045998.0,
	"step": 484
	},
	{
	"epoch": 1.7160740002215575,
	"grad_norm": 0.08344466580301783,
	"learning_rate": 5.945547902620067e-06,
	"loss": 0.1159,
	"num_tokens": 2011230617.0,
	"step": 485
	},
	{
	"epoch": 1.7196189210147335,
	"grad_norm": 0.07243482317247132,
	"learning_rate": 5.899232519881684e-06,
	"loss": 0.1145,
	"num_tokens": 2015399820.0,
	"step": 486
	},
	{
	"epoch": 1.7231638418079096,
	"grad_norm": 0.07890421658581434,
	"learning_rate": 5.85344436755047e-06,
	"loss": 0.1182,
	"num_tokens": 2019555573.0,
	"step": 487
	},
	{
	"epoch": 1.7267087626010857,
	"grad_norm": 0.08782878255230156,
	"learning_rate": 5.8081849449901215e-06,
	"loss": 0.1174,
	"num_tokens": 2023680879.0,
	"step": 488
	},
	{
	"epoch": 1.7302536833942617,
	"grad_norm": 0.07559306230279811,
	"learning_rate": 5.763455734250729e-06,
	"loss": 0.1158,
	"num_tokens": 2027814121.0,
	"step": 489
	},
	{
	"epoch": 1.7337986041874376,
	"grad_norm": 0.08008010405949205,
	"learning_rate": 5.719258200020226e-06,
	"loss": 0.1212,
	"num_tokens": 2031993234.0,
	"step": 490
	},
	{
	"epoch": 1.7373435249806137,
	"grad_norm": 0.07649091594007568,
	"learning_rate": 5.675593789576453e-06,
	"loss": 0.1166,
	"num_tokens": 2036152471.0,
	"step": 491
	},
	{
	"epoch": 1.7408884457737899,
	"grad_norm": 0.08242828819484474,
	"learning_rate": 5.632463932739758e-06,
	"loss": 0.1162,
	"num_tokens": 2040328672.0,
	"step": 492
	},
	{
	"epoch": 1.7444333665669658,
	"grad_norm": 0.0764779758520097,
	"learning_rate": 5.5898700418261634e-06,
	"loss": 0.1136,
	"num_tokens": 2044469128.0,
	"step": 493
	},
	{
	"epoch": 1.7479782873601417,
	"grad_norm": 0.08082082201817449,
	"learning_rate": 5.547813511601141e-06,
	"loss": 0.1182,
	"num_tokens": 2048638492.0,
	"step": 494
	},
	{
	"epoch": 1.7515232081533179,
	"grad_norm": 0.08672851884247582,
	"learning_rate": 5.506295719233917e-06,
	"loss": 0.1126,
	"num_tokens": 2052766361.0,
	"step": 495
	},
	{
	"epoch": 1.755068128946494,
	"grad_norm": 0.07708530243564624,
	"learning_rate": 5.465318024252388e-06,
	"loss": 0.1118,
	"num_tokens": 2056923330.0,
	"step": 496
	},
	{
	"epoch": 1.75861304973967,
	"grad_norm": 0.08249918638019296,
	"learning_rate": 5.424881768498606e-06,
	"loss": 0.1184,
	"num_tokens": 2061082608.0,
	"step": 497
	},
	{
	"epoch": 1.7621579705328458,
	"grad_norm": 0.07528201853205667,
	"learning_rate": 5.384988276084832e-06,
	"loss": 0.1154,
	"num_tokens": 2065215670.0,
	"step": 498
	},
	{
	"epoch": 1.765702891326022,
	"grad_norm": 0.0828797901951655,
	"learning_rate": 5.3456388533501705e-06,
	"loss": 0.1177,
	"num_tokens": 2069337092.0,
	"step": 499
	},
	{
	"epoch": 1.7692478121191981,
	"grad_norm": 0.07757896704821854,
	"learning_rate": 5.3068347888178e-06,
	"loss": 0.1193,
	"num_tokens": 2073503065.0,
	"step": 500
	},
	{
	"epoch": 1.7727927329123738,
	"grad_norm": 0.08719904171754243,
	"learning_rate": 5.268577353152797e-06,
	"loss": 0.1133,
	"num_tokens": 2077644951.0,
	"step": 501
	},
	{
	"epoch": 1.77633765370555,
	"grad_norm": 0.0753761856514278,
	"learning_rate": 5.230867799120486e-06,
	"loss": 0.1209,
	"num_tokens": 2081775370.0,
	"step": 502
	},
	{
	"epoch": 1.7798825744987261,
	"grad_norm": 0.07650652786309468,
	"learning_rate": 5.193707361545463e-06,
	"loss": 0.1136,
	"num_tokens": 2085898256.0,
	"step": 503
	},
	{
	"epoch": 1.783427495291902,
	"grad_norm": 0.08169803036886576,
	"learning_rate": 5.157097257271117e-06,
	"loss": 0.1163,
	"num_tokens": 2090053566.0,
	"step": 504
	},
	{
	"epoch": 1.786972416085078,
	"grad_norm": 0.07809954817657072,
	"learning_rate": 5.121038685119818e-06,
	"loss": 0.1145,
	"num_tokens": 2094195200.0,
	"step": 505
	},
	{
	"epoch": 1.790517336878254,
	"grad_norm": 0.08943493727645552,
	"learning_rate": 5.085532825853651e-06,
	"loss": 0.116,
	"num_tokens": 2098366980.0,
	"step": 506
	},
	{
	"epoch": 1.7940622576714302,
	"grad_norm": 0.07392171042310733,
	"learning_rate": 5.050580842135736e-06,
	"loss": 0.1189,
	"num_tokens": 2102537298.0,
	"step": 507
	},
	{
	"epoch": 1.7976071784646062,
	"grad_norm": 0.07492622977860645,
	"learning_rate": 5.01618387849217e-06,
	"loss": 0.1116,
	"num_tokens": 2106674584.0,
	"step": 508
	},
	{
	"epoch": 1.801152099257782,
	"grad_norm": 0.07820065869812615,
	"learning_rate": 4.982343061274557e-06,
	"loss": 0.1204,
	"num_tokens": 2110842861.0,
	"step": 509
	},
	{
	"epoch": 1.8046970200509582,
	"grad_norm": 0.07667183759716453,
	"learning_rate": 4.949059498623097e-06,
	"loss": 0.1183,
	"num_tokens": 2114962254.0,
	"step": 510
	},
	{
	"epoch": 1.8082419408441344,
	"grad_norm": 0.07759748538794982,
	"learning_rate": 4.916334280430333e-06,
	"loss": 0.1183,
	"num_tokens": 2119117319.0,
	"step": 511
	},
	{
	"epoch": 1.8117868616373103,
	"grad_norm": 0.07447007933945596,
	"learning_rate": 4.884168478305433e-06,
	"loss": 0.1169,
	"num_tokens": 2123260647.0,
	"step": 512
	},
	{
	"epoch": 1.8153317824304862,
	"grad_norm": 0.07708523048330068,
	"learning_rate": 4.852563145539116e-06,
	"loss": 0.1136,
	"num_tokens": 2127408620.0,
	"step": 513
	},
	{
	"epoch": 1.8188767032236623,
	"grad_norm": 0.07343966874592571,
	"learning_rate": 4.821519317069155e-06,
	"loss": 0.1169,
	"num_tokens": 2131570662.0,
	"step": 514
	},
	{
	"epoch": 1.8224216240168385,
	"grad_norm": 0.07872922867210017,
	"learning_rate": 4.791038009446497e-06,
	"loss": 0.1206,
	"num_tokens": 2135697205.0,
	"step": 515
	},
	{
	"epoch": 1.8259665448100144,
	"grad_norm": 0.07270179568376721,
	"learning_rate": 4.761120220801956e-06,
	"loss": 0.1134,
	"num_tokens": 2139839064.0,
	"step": 516
	},
	{
	"epoch": 1.8295114656031903,
	"grad_norm": 0.07496476854494366,
	"learning_rate": 4.731766930813543e-06,
	"loss": 0.1192,
	"num_tokens": 2143986628.0,
	"step": 517
	},
	{
	"epoch": 1.8330563863963665,
	"grad_norm": 0.0744096874470164,
	"learning_rate": 4.7029791006743895e-06,
	"loss": 0.1171,
	"num_tokens": 2148151849.0,
	"step": 518
	},
	{
	"epoch": 1.8366013071895426,
	"grad_norm": 0.079422884520098,
	"learning_rate": 4.674757673061255e-06,
	"loss": 0.1169,
	"num_tokens": 2152301466.0,
	"step": 519
	},
	{
	"epoch": 1.8401462279827185,
	"grad_norm": 0.07189245138812239,
	"learning_rate": 4.64710357210368e-06,
	"loss": 0.1146,
	"num_tokens": 2156445359.0,
	"step": 520
	},
	{
	"epoch": 1.8436911487758945,
	"grad_norm": 0.07695588530472214,
	"learning_rate": 4.620017703353707e-06,
	"loss": 0.1155,
	"num_tokens": 2160624764.0,
	"step": 521
	},
	{
	"epoch": 1.8472360695690706,
	"grad_norm": 0.07569831293614614,
	"learning_rate": 4.593500953756233e-06,
	"loss": 0.1181,
	"num_tokens": 2164786356.0,
	"step": 522
	},
	{
	"epoch": 1.8507809903622467,
	"grad_norm": 0.06592011885894174,
	"learning_rate": 4.567554191619979e-06,
	"loss": 0.117,
	"num_tokens": 2168954974.0,
	"step": 523
	},
	{
	"epoch": 1.8543259111554227,
	"grad_norm": 0.07023995487662163,
	"learning_rate": 4.5421782665890284e-06,
	"loss": 0.1192,
	"num_tokens": 2173114878.0,
	"step": 524
	},
	{
	"epoch": 1.8578708319485986,
	"grad_norm": 0.07128295559582981,
	"learning_rate": 4.517374009615037e-06,
	"loss": 0.118,
	"num_tokens": 2177293363.0,
	"step": 525
	},
	{
	"epoch": 1.8614157527417747,
	"grad_norm": 0.07437063386154072,
	"learning_rate": 4.493142232929995e-06,
	"loss": 0.118,
	"num_tokens": 2181422687.0,
	"step": 526
	},
	{
	"epoch": 1.8649606735349507,
	"grad_norm": 0.06995619144504099,
	"learning_rate": 4.46948373001965e-06,
	"loss": 0.1181,
	"num_tokens": 2185598154.0,
	"step": 527
	},
	{
	"epoch": 1.8685055943281266,
	"grad_norm": 0.07067192154362628,
	"learning_rate": 4.446399275597515e-06,
	"loss": 0.1177,
	"num_tokens": 2189768704.0,
	"step": 528
	},
	{
	"epoch": 1.8720505151213027,
	"grad_norm": 0.07922246482631316,
	"learning_rate": 4.423889625579501e-06,
	"loss": 0.1201,
	"num_tokens": 2193945580.0,
	"step": 529
	},
	{
	"epoch": 1.8755954359144789,
	"grad_norm": 0.07365230944828541,
	"learning_rate": 4.401955517059162e-06,
	"loss": 0.1127,
	"num_tokens": 2198106812.0,
	"step": 530
	},
	{
	"epoch": 1.8791403567076548,
	"grad_norm": 0.0759784617444428,
	"learning_rate": 4.38059766828356e-06,
	"loss": 0.1222,
	"num_tokens": 2202257695.0,
	"step": 531
	},
	{
	"epoch": 1.8826852775008307,
	"grad_norm": 0.07681519256346075,
	"learning_rate": 4.359816778629749e-06,
	"loss": 0.1079,
	"num_tokens": 2206390705.0,
	"step": 532
	},
	{
	"epoch": 1.8862301982940068,
	"grad_norm": 0.07509035728615902,
	"learning_rate": 4.33961352858187e-06,
	"loss": 0.1147,
	"num_tokens": 2210526109.0,
	"step": 533
	},
	{
	"epoch": 1.889775119087183,
	"grad_norm": 0.0733692236079307,
	"learning_rate": 4.3199885797088695e-06,
	"loss": 0.1115,
	"num_tokens": 2214680478.0,
	"step": 534
	},
	{
	"epoch": 1.893320039880359,
	"grad_norm": 0.07517249372356859,
	"learning_rate": 4.300942574642829e-06,
	"loss": 0.1125,
	"num_tokens": 2218834922.0,
	"step": 535
	},
	{
	"epoch": 1.8968649606735348,
	"grad_norm": 0.07565630715875153,
	"learning_rate": 4.282476137057933e-06,
	"loss": 0.1197,
	"num_tokens": 2222990936.0,
	"step": 536
	},
	{
	"epoch": 1.900409881466711,
	"grad_norm": 0.07137999884914838,
	"learning_rate": 4.2645898716500385e-06,
	"loss": 0.1106,
	"num_tokens": 2227154656.0,
	"step": 537
	},
	{
	"epoch": 1.9039548022598871,
	"grad_norm": 0.07768244537305824,
	"learning_rate": 4.24728436411688e-06,
	"loss": 0.1237,
	"num_tokens": 2231266480.0,
	"step": 538
	},
	{
	"epoch": 1.907499723053063,
	"grad_norm": 0.07481057078343745,
	"learning_rate": 4.230560181138884e-06,
	"loss": 0.1136,
	"num_tokens": 2235428917.0,
	"step": 539
	},
	{
	"epoch": 1.911044643846239,
	"grad_norm": 0.07201368345162525,
	"learning_rate": 4.214417870360614e-06,
	"loss": 0.1198,
	"num_tokens": 2239603436.0,
	"step": 540
	},
	{
	"epoch": 1.914589564639415,
	"grad_norm": 0.07469689369951103,
	"learning_rate": 4.198857960372837e-06,
	"loss": 0.1168,
	"num_tokens": 2243754477.0,
	"step": 541
	},
	{
	"epoch": 1.9181344854325912,
	"grad_norm": 0.07042106351849874,
	"learning_rate": 4.183880960695225e-06,
	"loss": 0.1169,
	"num_tokens": 2247923566.0,
	"step": 542
	},
	{
	"epoch": 1.9216794062257672,
	"grad_norm": 0.06971541992634243,
	"learning_rate": 4.169487361759655e-06,
	"loss": 0.1115,
	"num_tokens": 2252099431.0,
	"step": 543
	},
	{
	"epoch": 1.925224327018943,
	"grad_norm": 0.0699199526237033,
	"learning_rate": 4.155677634894157e-06,
	"loss": 0.1158,
	"num_tokens": 2256267131.0,
	"step": 544
	},
	{
	"epoch": 1.9287692478121192,
	"grad_norm": 0.06932261603891499,
	"learning_rate": 4.142452232307477e-06,
	"loss": 0.1169,
	"num_tokens": 2260438453.0,
	"step": 545
	},
	{
	"epoch": 1.9323141686052954,
	"grad_norm": 0.0702875655660896,
	"learning_rate": 4.129811587074279e-06,
	"loss": 0.1182,
	"num_tokens": 2264610517.0,
	"step": 546
	},
	{
	"epoch": 1.9358590893984713,
	"grad_norm": 0.06860901771471822,
	"learning_rate": 4.117756113120952e-06,
	"loss": 0.117,
	"num_tokens": 2268784580.0,
	"step": 547
	},
	{
	"epoch": 1.9394040101916472,
	"grad_norm": 0.07061101738960317,
	"learning_rate": 4.106286205212054e-06,
	"loss": 0.1229,
	"num_tokens": 2272938460.0,
	"step": 548
	},
	{
	"epoch": 1.9429489309848234,
	"grad_norm": 0.07008793261250126,
	"learning_rate": 4.0954022389373975e-06,
	"loss": 0.1178,
	"num_tokens": 2277096215.0,
	"step": 549
	},
	{
	"epoch": 1.9464938517779995,
	"grad_norm": 0.06765909382129992,
	"learning_rate": 4.085104570699745e-06,
	"loss": 0.1137,
	"num_tokens": 2281237511.0,
	"step": 550
	},
	{
	"epoch": 1.9500387725711754,
	"grad_norm": 0.07111557611031176,
	"learning_rate": 4.075393537703135e-06,
	"loss": 0.1221,
	"num_tokens": 2285402210.0,
	"step": 551
	},
	{
	"epoch": 1.9535836933643513,
	"grad_norm": 0.0719277282746577,
	"learning_rate": 4.0662694579418405e-06,
	"loss": 0.1134,
	"num_tokens": 2289555346.0,
	"step": 552
	},
	{
	"epoch": 1.9571286141575275,
	"grad_norm": 0.0746696683110134,
	"learning_rate": 4.057732630189961e-06,
	"loss": 0.1182,
	"num_tokens": 2293708211.0,
	"step": 553
	},
	{
	"epoch": 1.9606735349507034,
	"grad_norm": 0.06701318426114623,
	"learning_rate": 4.049783333991636e-06,
	"loss": 0.1114,
	"num_tokens": 2297854640.0,
	"step": 554
	},
	{
	"epoch": 1.9642184557438793,
	"grad_norm": 0.06669660787778318,
	"learning_rate": 4.042421829651887e-06,
	"loss": 0.1152,
	"num_tokens": 2302027463.0,
	"step": 555
	},
	{
	"epoch": 1.9677633765370555,
	"grad_norm": 0.06740724510480171,
	"learning_rate": 4.035648358228102e-06,
	"loss": 0.1181,
	"num_tokens": 2306182447.0,
	"step": 556
	},
	{
	"epoch": 1.9713082973302316,
	"grad_norm": 0.06790684700207919,
	"learning_rate": 4.029463141522137e-06,
	"loss": 0.1145,
	"num_tokens": 2310331979.0,
	"step": 557
	},
	{
	"epoch": 1.9748532181234075,
	"grad_norm": 0.06896307576095775,
	"learning_rate": 4.0238663820730515e-06,
	"loss": 0.1154,
	"num_tokens": 2314511002.0,
	"step": 558
	},
	{
	"epoch": 1.9783981389165834,
	"grad_norm": 0.06682786067741603,
	"learning_rate": 4.01885826315048e-06,
	"loss": 0.1172,
	"num_tokens": 2318681782.0,
	"step": 559
	},
	{
	"epoch": 1.9819430597097596,
	"grad_norm": 0.06680166505332281,
	"learning_rate": 4.014438948748629e-06,
	"loss": 0.1176,
	"num_tokens": 2322861630.0,
	"step": 560
	},
	{
	"epoch": 1.9854879805029357,
	"grad_norm": 0.06974773737262352,
	"learning_rate": 4.010608583580906e-06,
	"loss": 0.1154,
	"num_tokens": 2327027664.0,
	"step": 561
	},
	{
	"epoch": 1.9890329012961117,
	"grad_norm": 0.0633497863723213,
	"learning_rate": 4.007367293075181e-06,
	"loss": 0.1151,
	"num_tokens": 2331203870.0,
	"step": 562
	},
	{
	"epoch": 1.9925778220892876,
	"grad_norm": 0.07222338277634825,
	"learning_rate": 4.004715183369683e-06,
	"loss": 0.1164,
	"num_tokens": 2335365280.0,
	"step": 563
	},
	{
	"epoch": 1.9961227428824637,
	"grad_norm": 0.0709545855964077,
	"learning_rate": 4.0026523413095165e-06,
	"loss": 0.1179,
	"num_tokens": 2339520665.0,
	"step": 564
	},
	{
	"epoch": 1.9996676636756399,
	"grad_norm": 0.06889801876769293,
	"learning_rate": 4.001178834443828e-06,
	"loss": 0.1168,
	"num_tokens": 2343684842.0,
	"step": 565
	},
	{
	"epoch": 2.0,
	"grad_norm": 0.1861252601606322,
	"learning_rate": 4.000294711023586e-06,
	"loss": 0.1226,
	"num_tokens": 2343880620.0,
	"step": 566
	},
	{
	"epoch": 2.0,
	"step": 566,
	"total_flos": 1.0975557049441583e+19,
	"train_loss": 0.15245879367783718,
	"train_runtime": 38198.9028,
	"train_samples_per_second": 3.781,
	"train_steps_per_second": 0.015
	}
	],
	"logging_steps": 1,
	"max_steps": 566,
	"num_input_tokens_seen": 0,
	"num_train_epochs": 2,
	"save_steps": 200,
	"stateful_callbacks": {
	"TrainerControl": {
	"args": {
	"should_epoch_stop": false,
	"should_evaluate": false,
	"should_log": false,
	"should_save": true,
	"should_training_stop": true
	},
	"attributes": {}
	}
	},
	"total_flos": 1.0975557049441583e+19,
	"train_batch_size": 1,
	"trial_name": null,
	"trial_params": null
	}