Instructions to use formalmathatepfl/ApertusFinetuned02 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use formalmathatepfl/ApertusFinetuned02 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="formalmathatepfl/ApertusFinetuned02")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("formalmathatepfl/ApertusFinetuned02")
model = AutoModelForCausalLM.from_pretrained("formalmathatepfl/ApertusFinetuned02")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use formalmathatepfl/ApertusFinetuned02 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "formalmathatepfl/ApertusFinetuned02"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "formalmathatepfl/ApertusFinetuned02",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/formalmathatepfl/ApertusFinetuned02

SGLang

How to use formalmathatepfl/ApertusFinetuned02 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "formalmathatepfl/ApertusFinetuned02" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "formalmathatepfl/ApertusFinetuned02",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "formalmathatepfl/ApertusFinetuned02" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "formalmathatepfl/ApertusFinetuned02",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use formalmathatepfl/ApertusFinetuned02 with Docker Model Runner:
```
docker model run hf.co/formalmathatepfl/ApertusFinetuned02
```

ApertusFinetuned02 / trainer_state.json

raduv98

Upload folder using huggingface_hub

0152fcc verified 5 months ago

Raw

History Blame Contribute Delete

113 kB

	{
	"best_global_step": null,
	"best_metric": null,
	"best_model_checkpoint": null,
	"epoch": 1.0,
	"eval_steps": 500,
	"global_step": 12855,
	"is_hyper_param_search": false,
	"is_local_process_zero": true,
	"is_world_process_zero": true,
	"log_history": [
	{
	"epoch": 0.001555875374382512,
	"grad_norm": 17.319619719450433,
	"learning_rate": 4.922279792746115e-06,
	"loss": 1.1942,
	"step": 20
	},
	{
	"epoch": 0.003111750748765024,
	"grad_norm": 6.034311425292535,
	"learning_rate": 1.0103626943005182e-05,
	"loss": 0.6039,
	"step": 40
	},
	{
	"epoch": 0.004667626123147536,
	"grad_norm": 1.9730509967930208,
	"learning_rate": 1.528497409326425e-05,
	"loss": 0.3162,
	"step": 60
	},
	{
	"epoch": 0.006223501497530048,
	"grad_norm": 1.2032326466599368,
	"learning_rate": 2.046632124352332e-05,
	"loss": 0.2512,
	"step": 80
	},
	{
	"epoch": 0.00777937687191256,
	"grad_norm": 0.7629090226887217,
	"learning_rate": 2.5647668393782386e-05,
	"loss": 0.2,
	"step": 100
	},
	{
	"epoch": 0.009335252246295072,
	"grad_norm": 1.9755210522793252,
	"learning_rate": 3.082901554404145e-05,
	"loss": 0.2222,
	"step": 120
	},
	{
	"epoch": 0.010891127620677584,
	"grad_norm": 0.6247922085095571,
	"learning_rate": 3.601036269430052e-05,
	"loss": 0.2031,
	"step": 140
	},
	{
	"epoch": 0.012447002995060095,
	"grad_norm": 0.9921659226449822,
	"learning_rate": 4.119170984455959e-05,
	"loss": 0.1727,
	"step": 160
	},
	{
	"epoch": 0.014002878369442608,
	"grad_norm": 1.5274033440451522,
	"learning_rate": 4.637305699481865e-05,
	"loss": 0.1959,
	"step": 180
	},
	{
	"epoch": 0.01555875374382512,
	"grad_norm": 1.071285286348304,
	"learning_rate": 5.155440414507773e-05,
	"loss": 0.2178,
	"step": 200
	},
	{
	"epoch": 0.01711462911820763,
	"grad_norm": 0.6532820982550468,
	"learning_rate": 5.673575129533679e-05,
	"loss": 0.1624,
	"step": 220
	},
	{
	"epoch": 0.018670504492590143,
	"grad_norm": 1.318545012052198,
	"learning_rate": 6.191709844559586e-05,
	"loss": 0.2156,
	"step": 240
	},
	{
	"epoch": 0.020226379866972656,
	"grad_norm": 0.5354893743157717,
	"learning_rate": 6.709844559585493e-05,
	"loss": 0.206,
	"step": 260
	},
	{
	"epoch": 0.02178225524135517,
	"grad_norm": 0.7031058657181408,
	"learning_rate": 7.2279792746114e-05,
	"loss": 0.2041,
	"step": 280
	},
	{
	"epoch": 0.023338130615737678,
	"grad_norm": 0.6792963306860744,
	"learning_rate": 7.746113989637306e-05,
	"loss": 0.2034,
	"step": 300
	},
	{
	"epoch": 0.02489400599012019,
	"grad_norm": 0.6950644209993052,
	"learning_rate": 8.264248704663214e-05,
	"loss": 0.1893,
	"step": 320
	},
	{
	"epoch": 0.026449881364502704,
	"grad_norm": 0.6792296561480745,
	"learning_rate": 8.78238341968912e-05,
	"loss": 0.1866,
	"step": 340
	},
	{
	"epoch": 0.028005756738885217,
	"grad_norm": 0.8438742197912741,
	"learning_rate": 9.300518134715026e-05,
	"loss": 0.1914,
	"step": 360
	},
	{
	"epoch": 0.029561632113267726,
	"grad_norm": 0.666058547279043,
	"learning_rate": 9.818652849740934e-05,
	"loss": 0.1942,
	"step": 380
	},
	{
	"epoch": 0.03111750748765024,
	"grad_norm": 0.5926641030552818,
	"learning_rate": 9.999973179750075e-05,
	"loss": 0.1995,
	"step": 400
	},
	{
	"epoch": 0.03267338286203275,
	"grad_norm": 1.614365091067381,
	"learning_rate": 9.999827176863762e-05,
	"loss": 0.1975,
	"step": 420
	},
	{
	"epoch": 0.03422925823641526,
	"grad_norm": 1.2170992115068964,
	"learning_rate": 9.999554218757943e-05,
	"loss": 0.2005,
	"step": 440
	},
	{
	"epoch": 0.03578513361079778,
	"grad_norm": 0.43229633955852237,
	"learning_rate": 9.999154312363545e-05,
	"loss": 0.1703,
	"step": 460
	},
	{
	"epoch": 0.037341008985180286,
	"grad_norm": 1.1327890899304212,
	"learning_rate": 9.99862746783496e-05,
	"loss": 0.1946,
	"step": 480
	},
	{
	"epoch": 0.038896884359562796,
	"grad_norm": 0.32826735454181755,
	"learning_rate": 9.997973698549783e-05,
	"loss": 0.1749,
	"step": 500
	},
	{
	"epoch": 0.04045275973394531,
	"grad_norm": 0.7176353061623324,
	"learning_rate": 9.997193021108474e-05,
	"loss": 0.1828,
	"step": 520
	},
	{
	"epoch": 0.04200863510832782,
	"grad_norm": 0.4965966286987883,
	"learning_rate": 9.996285455333931e-05,
	"loss": 0.2116,
	"step": 540
	},
	{
	"epoch": 0.04356451048271034,
	"grad_norm": 0.5586409398977028,
	"learning_rate": 9.995251024270995e-05,
	"loss": 0.1749,
	"step": 560
	},
	{
	"epoch": 0.04512038585709285,
	"grad_norm": 0.7023030858646293,
	"learning_rate": 9.994089754185858e-05,
	"loss": 0.1914,
	"step": 580
	},
	{
	"epoch": 0.046676261231475356,
	"grad_norm": 1.1878929271237093,
	"learning_rate": 9.992801674565399e-05,
	"loss": 0.2059,
	"step": 600
	},
	{
	"epoch": 0.04823213660585787,
	"grad_norm": 0.3524647362644913,
	"learning_rate": 9.991386818116433e-05,
	"loss": 0.1753,
	"step": 620
	},
	{
	"epoch": 0.04978801198024038,
	"grad_norm": 0.7634525063604128,
	"learning_rate": 9.989845220764885e-05,
	"loss": 0.178,
	"step": 640
	},
	{
	"epoch": 0.05134388735462289,
	"grad_norm": 0.42869001265806606,
	"learning_rate": 9.988176921654875e-05,
	"loss": 0.2126,
	"step": 660
	},
	{
	"epoch": 0.05289976272900541,
	"grad_norm": 0.8394630140106453,
	"learning_rate": 9.986381963147722e-05,
	"loss": 0.1812,
	"step": 680
	},
	{
	"epoch": 0.05445563810338792,
	"grad_norm": 0.37590180288467034,
	"learning_rate": 9.984460390820873e-05,
	"loss": 0.1902,
	"step": 700
	},
	{
	"epoch": 0.05601151347777043,
	"grad_norm": 0.6291695124948697,
	"learning_rate": 9.98241225346674e-05,
	"loss": 0.1958,
	"step": 720
	},
	{
	"epoch": 0.05756738885215294,
	"grad_norm": 0.6795414979844046,
	"learning_rate": 9.980237603091466e-05,
	"loss": 0.1756,
	"step": 740
	},
	{
	"epoch": 0.05912326422653545,
	"grad_norm": 0.8039594905278015,
	"learning_rate": 9.977936494913605e-05,
	"loss": 0.1766,
	"step": 760
	},
	{
	"epoch": 0.06067913960091797,
	"grad_norm": 0.662397405104178,
	"learning_rate": 9.975508987362715e-05,
	"loss": 0.1983,
	"step": 780
	},
	{
	"epoch": 0.06223501497530048,
	"grad_norm": 0.7130257099924733,
	"learning_rate": 9.972955142077874e-05,
	"loss": 0.1431,
	"step": 800
	},
	{
	"epoch": 0.063790890349683,
	"grad_norm": 0.39998537992243943,
	"learning_rate": 9.970275023906125e-05,
	"loss": 0.164,
	"step": 820
	},
	{
	"epoch": 0.0653467657240655,
	"grad_norm": 0.31784157070733676,
	"learning_rate": 9.967468700900816e-05,
	"loss": 0.1387,
	"step": 840
	},
	{
	"epoch": 0.06690264109844801,
	"grad_norm": 0.432463533713304,
	"learning_rate": 9.964536244319882e-05,
	"loss": 0.1609,
	"step": 860
	},
	{
	"epoch": 0.06845851647283052,
	"grad_norm": 0.2763686518835813,
	"learning_rate": 9.96147772862403e-05,
	"loss": 0.1602,
	"step": 880
	},
	{
	"epoch": 0.07001439184721304,
	"grad_norm": 0.9616104159797957,
	"learning_rate": 9.95829323147485e-05,
	"loss": 0.1736,
	"step": 900
	},
	{
	"epoch": 0.07157026722159555,
	"grad_norm": 0.48412231508727205,
	"learning_rate": 9.954982833732845e-05,
	"loss": 0.169,
	"step": 920
	},
	{
	"epoch": 0.07312614259597806,
	"grad_norm": 0.5796742391706576,
	"learning_rate": 9.951546619455374e-05,
	"loss": 0.1802,
	"step": 940
	},
	{
	"epoch": 0.07468201797036057,
	"grad_norm": 1.9518841112093603,
	"learning_rate": 9.947984675894522e-05,
	"loss": 0.1552,
	"step": 960
	},
	{
	"epoch": 0.07623789334474308,
	"grad_norm": 0.466369736705894,
	"learning_rate": 9.944297093494877e-05,
	"loss": 0.1813,
	"step": 980
	},
	{
	"epoch": 0.07779376871912559,
	"grad_norm": 0.7276335450577125,
	"learning_rate": 9.940483965891247e-05,
	"loss": 0.1481,
	"step": 1000
	},
	{
	"epoch": 0.07934964409350811,
	"grad_norm": 0.43517555855293155,
	"learning_rate": 9.936545389906264e-05,
	"loss": 0.1549,
	"step": 1020
	},
	{
	"epoch": 0.08090551946789062,
	"grad_norm": 0.6217227948010409,
	"learning_rate": 9.932481465547944e-05,
	"loss": 0.1732,
	"step": 1040
	},
	{
	"epoch": 0.08246139484227313,
	"grad_norm": 0.39284540496659015,
	"learning_rate": 9.928292296007133e-05,
	"loss": 0.1667,
	"step": 1060
	},
	{
	"epoch": 0.08401727021665564,
	"grad_norm": 0.4181589067587452,
	"learning_rate": 9.923977987654899e-05,
	"loss": 0.1618,
	"step": 1080
	},
	{
	"epoch": 0.08557314559103815,
	"grad_norm": 0.34195461831480034,
	"learning_rate": 9.919538650039818e-05,
	"loss": 0.1809,
	"step": 1100
	},
	{
	"epoch": 0.08712902096542068,
	"grad_norm": 0.3083286930380637,
	"learning_rate": 9.914974395885202e-05,
	"loss": 0.1781,
	"step": 1120
	},
	{
	"epoch": 0.08868489633980318,
	"grad_norm": 0.5269178730421432,
	"learning_rate": 9.910285341086238e-05,
	"loss": 0.1497,
	"step": 1140
	},
	{
	"epoch": 0.0902407717141857,
	"grad_norm": 0.581918599545614,
	"learning_rate": 9.905471604707036e-05,
	"loss": 0.1627,
	"step": 1160
	},
	{
	"epoch": 0.0917966470885682,
	"grad_norm": 0.39860799455821855,
	"learning_rate": 9.90053330897761e-05,
	"loss": 0.1566,
	"step": 1180
	},
	{
	"epoch": 0.09335252246295071,
	"grad_norm": 0.32373325029140804,
	"learning_rate": 9.895470579290782e-05,
	"loss": 0.1627,
	"step": 1200
	},
	{
	"epoch": 0.09490839783733324,
	"grad_norm": 0.5937577484229394,
	"learning_rate": 9.890283544198984e-05,
	"loss": 0.157,
	"step": 1220
	},
	{
	"epoch": 0.09646427321171575,
	"grad_norm": 0.34588234591735073,
	"learning_rate": 9.884972335411006e-05,
	"loss": 0.158,
	"step": 1240
	},
	{
	"epoch": 0.09802014858609825,
	"grad_norm": 1.2341189236115186,
	"learning_rate": 9.879537087788644e-05,
	"loss": 0.1546,
	"step": 1260
	},
	{
	"epoch": 0.09957602396048076,
	"grad_norm": 0.397736624317743,
	"learning_rate": 9.873977939343281e-05,
	"loss": 0.1565,
	"step": 1280
	},
	{
	"epoch": 0.10113189933486327,
	"grad_norm": 0.47140719304539397,
	"learning_rate": 9.868295031232376e-05,
	"loss": 0.1619,
	"step": 1300
	},
	{
	"epoch": 0.10268777470924578,
	"grad_norm": 0.5541168503508661,
	"learning_rate": 9.862488507755887e-05,
	"loss": 0.1718,
	"step": 1320
	},
	{
	"epoch": 0.1042436500836283,
	"grad_norm": 0.397811698688364,
	"learning_rate": 9.856558516352604e-05,
	"loss": 0.1762,
	"step": 1340
	},
	{
	"epoch": 0.10579952545801082,
	"grad_norm": 0.2854618515682191,
	"learning_rate": 9.850505207596399e-05,
	"loss": 0.1557,
	"step": 1360
	},
	{
	"epoch": 0.10735540083239332,
	"grad_norm": 0.4540824994740618,
	"learning_rate": 9.844328735192419e-05,
	"loss": 0.162,
	"step": 1380
	},
	{
	"epoch": 0.10891127620677583,
	"grad_norm": 0.685488162170286,
	"learning_rate": 9.838029255973164e-05,
	"loss": 0.1676,
	"step": 1400
	},
	{
	"epoch": 0.11046715158115834,
	"grad_norm": 0.5117834764278872,
	"learning_rate": 9.831606929894515e-05,
	"loss": 0.164,
	"step": 1420
	},
	{
	"epoch": 0.11202302695554087,
	"grad_norm": 0.6165689497044521,
	"learning_rate": 9.825061920031676e-05,
	"loss": 0.1653,
	"step": 1440
	},
	{
	"epoch": 0.11357890232992338,
	"grad_norm": 0.44081742735619694,
	"learning_rate": 9.818394392575019e-05,
	"loss": 0.1712,
	"step": 1460
	},
	{
	"epoch": 0.11513477770430589,
	"grad_norm": 0.3893309898787136,
	"learning_rate": 9.81160451682588e-05,
	"loss": 0.1331,
	"step": 1480
	},
	{
	"epoch": 0.1166906530786884,
	"grad_norm": 0.8680569773139007,
	"learning_rate": 9.804692465192255e-05,
	"loss": 0.145,
	"step": 1500
	},
	{
	"epoch": 0.1182465284530709,
	"grad_norm": 0.47648960793193007,
	"learning_rate": 9.797658413184413e-05,
	"loss": 0.1537,
	"step": 1520
	},
	{
	"epoch": 0.11980240382745343,
	"grad_norm": 0.3548830715285749,
	"learning_rate": 9.790502539410451e-05,
	"loss": 0.124,
	"step": 1540
	},
	{
	"epoch": 0.12135827920183594,
	"grad_norm": 0.29309164999669174,
	"learning_rate": 9.783225025571758e-05,
	"loss": 0.1762,
	"step": 1560
	},
	{
	"epoch": 0.12291415457621845,
	"grad_norm": 0.34709843298560705,
	"learning_rate": 9.77582605645839e-05,
	"loss": 0.1615,
	"step": 1580
	},
	{
	"epoch": 0.12447002995060095,
	"grad_norm": 0.34961769775064666,
	"learning_rate": 9.768305819944393e-05,
	"loss": 0.1511,
	"step": 1600
	},
	{
	"epoch": 0.12602590532498348,
	"grad_norm": 0.4347111315984819,
	"learning_rate": 9.760664506983022e-05,
	"loss": 0.1478,
	"step": 1620
	},
	{
	"epoch": 0.127581780699366,
	"grad_norm": 0.4890449349693539,
	"learning_rate": 9.752902311601897e-05,
	"loss": 0.1552,
	"step": 1640
	},
	{
	"epoch": 0.1291376560737485,
	"grad_norm": 0.4695674637829446,
	"learning_rate": 9.745019430898074e-05,
	"loss": 0.182,
	"step": 1660
	},
	{
	"epoch": 0.130693531448131,
	"grad_norm": 0.4021355001375143,
	"learning_rate": 9.737016065033043e-05,
	"loss": 0.1624,
	"step": 1680
	},
	{
	"epoch": 0.13224940682251352,
	"grad_norm": 0.4101377747072328,
	"learning_rate": 9.728892417227638e-05,
	"loss": 0.1472,
	"step": 1700
	},
	{
	"epoch": 0.13380528219689602,
	"grad_norm": 0.4874566830624942,
	"learning_rate": 9.720648693756891e-05,
	"loss": 0.1568,
	"step": 1720
	},
	{
	"epoch": 0.13536115757127853,
	"grad_norm": 0.3070141026680839,
	"learning_rate": 9.71228510394478e-05,
	"loss": 0.1174,
	"step": 1740
	},
	{
	"epoch": 0.13691703294566104,
	"grad_norm": 0.7422243908450862,
	"learning_rate": 9.703801860158922e-05,
	"loss": 0.1624,
	"step": 1760
	},
	{
	"epoch": 0.13847290832004355,
	"grad_norm": 0.3359230971485835,
	"learning_rate": 9.695199177805177e-05,
	"loss": 0.1521,
	"step": 1780
	},
	{
	"epoch": 0.1400287836944261,
	"grad_norm": 0.34172790249454854,
	"learning_rate": 9.686477275322183e-05,
	"loss": 0.1283,
	"step": 1800
	},
	{
	"epoch": 0.1415846590688086,
	"grad_norm": 0.3183023197046218,
	"learning_rate": 9.6776363741758e-05,
	"loss": 0.1747,
	"step": 1820
	},
	{
	"epoch": 0.1431405344431911,
	"grad_norm": 0.3274803703619125,
	"learning_rate": 9.668676698853501e-05,
	"loss": 0.1488,
	"step": 1840
	},
	{
	"epoch": 0.14469640981757362,
	"grad_norm": 0.40155153378027947,
	"learning_rate": 9.659598476858653e-05,
	"loss": 0.1447,
	"step": 1860
	},
	{
	"epoch": 0.14625228519195613,
	"grad_norm": 0.42210540205654484,
	"learning_rate": 9.650401938704763e-05,
	"loss": 0.1508,
	"step": 1880
	},
	{
	"epoch": 0.14780816056633864,
	"grad_norm": 0.44393838217184634,
	"learning_rate": 9.641087317909602e-05,
	"loss": 0.1389,
	"step": 1900
	},
	{
	"epoch": 0.14936403594072115,
	"grad_norm": 0.37832221345056716,
	"learning_rate": 9.631654850989295e-05,
	"loss": 0.158,
	"step": 1920
	},
	{
	"epoch": 0.15091991131510366,
	"grad_norm": 0.8375668027000704,
	"learning_rate": 9.622104777452297e-05,
	"loss": 0.1504,
	"step": 1940
	},
	{
	"epoch": 0.15247578668948616,
	"grad_norm": 0.6433924231642061,
	"learning_rate": 9.61243733979333e-05,
	"loss": 0.1418,
	"step": 1960
	},
	{
	"epoch": 0.15403166206386867,
	"grad_norm": 0.5840426093998093,
	"learning_rate": 9.602652783487212e-05,
	"loss": 0.1569,
	"step": 1980
	},
	{
	"epoch": 0.15558753743825118,
	"grad_norm": 0.2782123166957972,
	"learning_rate": 9.592751356982628e-05,
	"loss": 0.136,
	"step": 2000
	},
	{
	"epoch": 0.15714341281263372,
	"grad_norm": 0.25122230701120113,
	"learning_rate": 9.582733311695824e-05,
	"loss": 0.1312,
	"step": 2020
	},
	{
	"epoch": 0.15869928818701623,
	"grad_norm": 0.2928464387893248,
	"learning_rate": 9.572598902004221e-05,
	"loss": 0.1411,
	"step": 2040
	},
	{
	"epoch": 0.16025516356139874,
	"grad_norm": 0.2524519704187489,
	"learning_rate": 9.562348385239953e-05,
	"loss": 0.1497,
	"step": 2060
	},
	{
	"epoch": 0.16181103893578125,
	"grad_norm": 0.5351271665666959,
	"learning_rate": 9.55198202168334e-05,
	"loss": 0.146,
	"step": 2080
	},
	{
	"epoch": 0.16336691431016376,
	"grad_norm": 0.32289788201801034,
	"learning_rate": 9.54150007455627e-05,
	"loss": 0.1417,
	"step": 2100
	},
	{
	"epoch": 0.16492278968454627,
	"grad_norm": 0.27821022441852955,
	"learning_rate": 9.530902810015522e-05,
	"loss": 0.1467,
	"step": 2120
	},
	{
	"epoch": 0.16647866505892878,
	"grad_norm": 0.2555525964943725,
	"learning_rate": 9.520190497146011e-05,
	"loss": 0.1538,
	"step": 2140
	},
	{
	"epoch": 0.16803454043331129,
	"grad_norm": 0.44356603181735377,
	"learning_rate": 9.50936340795394e-05,
	"loss": 0.1439,
	"step": 2160
	},
	{
	"epoch": 0.1695904158076938,
	"grad_norm": 0.40525490281044213,
	"learning_rate": 9.49842181735991e-05,
	"loss": 0.1601,
	"step": 2180
	},
	{
	"epoch": 0.1711462911820763,
	"grad_norm": 0.6236847200299162,
	"learning_rate": 9.487366003191931e-05,
	"loss": 0.1691,
	"step": 2200
	},
	{
	"epoch": 0.17270216655645884,
	"grad_norm": 0.27012006423017526,
	"learning_rate": 9.47619624617837e-05,
	"loss": 0.1424,
	"step": 2220
	},
	{
	"epoch": 0.17425804193084135,
	"grad_norm": 0.488179121288762,
	"learning_rate": 9.46491282994082e-05,
	"loss": 0.1611,
	"step": 2240
	},
	{
	"epoch": 0.17581391730522386,
	"grad_norm": 0.43056234190137654,
	"learning_rate": 9.453516040986902e-05,
	"loss": 0.153,
	"step": 2260
	},
	{
	"epoch": 0.17736979267960637,
	"grad_norm": 0.46418580767399376,
	"learning_rate": 9.442006168702987e-05,
	"loss": 0.1704,
	"step": 2280
	},
	{
	"epoch": 0.17892566805398888,
	"grad_norm": 0.3967935603456352,
	"learning_rate": 9.430383505346848e-05,
	"loss": 0.1677,
	"step": 2300
	},
	{
	"epoch": 0.1804815434283714,
	"grad_norm": 0.3554389824348632,
	"learning_rate": 9.418648346040243e-05,
	"loss": 0.155,
	"step": 2320
	},
	{
	"epoch": 0.1820374188027539,
	"grad_norm": 0.3290651460687597,
	"learning_rate": 9.406800988761415e-05,
	"loss": 0.1455,
	"step": 2340
	},
	{
	"epoch": 0.1835932941771364,
	"grad_norm": 0.4565651979760447,
	"learning_rate": 9.394841734337533e-05,
	"loss": 0.1528,
	"step": 2360
	},
	{
	"epoch": 0.18514916955151892,
	"grad_norm": 0.2965206639913831,
	"learning_rate": 9.382770886437045e-05,
	"loss": 0.1494,
	"step": 2380
	},
	{
	"epoch": 0.18670504492590143,
	"grad_norm": 0.4935731477937587,
	"learning_rate": 9.370588751561976e-05,
	"loss": 0.1385,
	"step": 2400
	},
	{
	"epoch": 0.18826092030028393,
	"grad_norm": 23.228259606683267,
	"learning_rate": 9.358295639040138e-05,
	"loss": 0.1477,
	"step": 2420
	},
	{
	"epoch": 0.18981679567466647,
	"grad_norm": 0.29775979144360015,
	"learning_rate": 9.345891861017279e-05,
	"loss": 0.1532,
	"step": 2440
	},
	{
	"epoch": 0.19137267104904898,
	"grad_norm": 0.5180073717771124,
	"learning_rate": 9.33337773244916e-05,
	"loss": 0.1834,
	"step": 2460
	},
	{
	"epoch": 0.1929285464234315,
	"grad_norm": 0.26816341301267577,
	"learning_rate": 9.320753571093551e-05,
	"loss": 0.1656,
	"step": 2480
	},
	{
	"epoch": 0.194484421797814,
	"grad_norm": 0.42034706047418385,
	"learning_rate": 9.308019697502167e-05,
	"loss": 0.1296,
	"step": 2500
	},
	{
	"epoch": 0.1960402971721965,
	"grad_norm": 0.2907014171256684,
	"learning_rate": 9.295176435012526e-05,
	"loss": 0.1395,
	"step": 2520
	},
	{
	"epoch": 0.19759617254657902,
	"grad_norm": 0.29190040680560275,
	"learning_rate": 9.282224109739746e-05,
	"loss": 0.1366,
	"step": 2540
	},
	{
	"epoch": 0.19915204792096153,
	"grad_norm": 0.22793845427263545,
	"learning_rate": 9.26916305056825e-05,
	"loss": 0.1402,
	"step": 2560
	},
	{
	"epoch": 0.20070792329534404,
	"grad_norm": 0.5677019435762835,
	"learning_rate": 9.255993589143429e-05,
	"loss": 0.1752,
	"step": 2580
	},
	{
	"epoch": 0.20226379866972655,
	"grad_norm": 0.24237173890462105,
	"learning_rate": 9.242716059863216e-05,
	"loss": 0.1613,
	"step": 2600
	},
	{
	"epoch": 0.20381967404410906,
	"grad_norm": 0.3916749839950002,
	"learning_rate": 9.229330799869592e-05,
	"loss": 0.1546,
	"step": 2620
	},
	{
	"epoch": 0.20537554941849157,
	"grad_norm": 0.5667574935919,
	"learning_rate": 9.215838149040029e-05,
	"loss": 0.1911,
	"step": 2640
	},
	{
	"epoch": 0.2069314247928741,
	"grad_norm": 0.42445850795345075,
	"learning_rate": 9.202238449978857e-05,
	"loss": 0.1803,
	"step": 2660
	},
	{
	"epoch": 0.2084873001672566,
	"grad_norm": 0.44930065129478325,
	"learning_rate": 9.18853204800857e-05,
	"loss": 0.1505,
	"step": 2680
	},
	{
	"epoch": 0.21004317554163912,
	"grad_norm": 0.3259924668841964,
	"learning_rate": 9.174719291161052e-05,
	"loss": 0.122,
	"step": 2700
	},
	{
	"epoch": 0.21159905091602163,
	"grad_norm": 0.30286492475849147,
	"learning_rate": 9.16080053016874e-05,
	"loss": 0.1357,
	"step": 2720
	},
	{
	"epoch": 0.21315492629040414,
	"grad_norm": 0.35290421972710057,
	"learning_rate": 9.146776118455722e-05,
	"loss": 0.14,
	"step": 2740
	},
	{
	"epoch": 0.21471080166478665,
	"grad_norm": 0.31530493548512184,
	"learning_rate": 9.132646412128761e-05,
	"loss": 0.1389,
	"step": 2760
	},
	{
	"epoch": 0.21626667703916916,
	"grad_norm": 0.41908000642474263,
	"learning_rate": 9.118411769968252e-05,
	"loss": 0.1625,
	"step": 2780
	},
	{
	"epoch": 0.21782255241355167,
	"grad_norm": 0.2993847704027942,
	"learning_rate": 9.10407255341911e-05,
	"loss": 0.1412,
	"step": 2800
	},
	{
	"epoch": 0.21937842778793418,
	"grad_norm": 0.22034475594395644,
	"learning_rate": 9.089629126581597e-05,
	"loss": 0.1668,
	"step": 2820
	},
	{
	"epoch": 0.2209343031623167,
	"grad_norm": 0.3800273519301784,
	"learning_rate": 9.075081856202076e-05,
	"loss": 0.1543,
	"step": 2840
	},
	{
	"epoch": 0.22249017853669922,
	"grad_norm": 0.256006679212103,
	"learning_rate": 9.060431111663696e-05,
	"loss": 0.1231,
	"step": 2860
	},
	{
	"epoch": 0.22404605391108173,
	"grad_norm": 0.2698578802702728,
	"learning_rate": 9.045677264977011e-05,
	"loss": 0.1422,
	"step": 2880
	},
	{
	"epoch": 0.22560192928546424,
	"grad_norm": 0.31556571339639367,
	"learning_rate": 9.030820690770541e-05,
	"loss": 0.1393,
	"step": 2900
	},
	{
	"epoch": 0.22715780465984675,
	"grad_norm": 0.2835658007302944,
	"learning_rate": 9.015861766281251e-05,
	"loss": 0.1397,
	"step": 2920
	},
	{
	"epoch": 0.22871368003422926,
	"grad_norm": 0.2848434185927417,
	"learning_rate": 9.00080087134498e-05,
	"loss": 0.1487,
	"step": 2940
	},
	{
	"epoch": 0.23026955540861177,
	"grad_norm": 0.20434109203915374,
	"learning_rate": 8.98563838838679e-05,
	"loss": 0.1381,
	"step": 2960
	},
	{
	"epoch": 0.23182543078299428,
	"grad_norm": 0.2684569544268393,
	"learning_rate": 8.970374702411259e-05,
	"loss": 0.1241,
	"step": 2980
	},
	{
	"epoch": 0.2333813061573768,
	"grad_norm": 0.4302874869501943,
	"learning_rate": 8.955010200992704e-05,
	"loss": 0.1675,
	"step": 3000
	},
	{
	"epoch": 0.2349371815317593,
	"grad_norm": 0.47404364846940084,
	"learning_rate": 8.939545274265339e-05,
	"loss": 0.1497,
	"step": 3020
	},
	{
	"epoch": 0.2364930569061418,
	"grad_norm": 0.5017915650073008,
	"learning_rate": 8.923980314913374e-05,
	"loss": 0.1353,
	"step": 3040
	},
	{
	"epoch": 0.23804893228052432,
	"grad_norm": 0.295380936064937,
	"learning_rate": 8.908315718161034e-05,
	"loss": 0.1345,
	"step": 3060
	},
	{
	"epoch": 0.23960480765490685,
	"grad_norm": 1.4800300631279193,
	"learning_rate": 8.892551881762527e-05,
	"loss": 0.1556,
	"step": 3080
	},
	{
	"epoch": 0.24116068302928936,
	"grad_norm": 0.3838493898052549,
	"learning_rate": 8.876689205991957e-05,
	"loss": 0.137,
	"step": 3100
	},
	{
	"epoch": 0.24271655840367187,
	"grad_norm": 0.3004585726966108,
	"learning_rate": 8.86072809363314e-05,
	"loss": 0.1371,
	"step": 3120
	},
	{
	"epoch": 0.24427243377805438,
	"grad_norm": 0.3112409549468866,
	"learning_rate": 8.844668949969389e-05,
	"loss": 0.1296,
	"step": 3140
	},
	{
	"epoch": 0.2458283091524369,
	"grad_norm": 0.423284480970118,
	"learning_rate": 8.828512182773219e-05,
	"loss": 0.1487,
	"step": 3160
	},
	{
	"epoch": 0.2473841845268194,
	"grad_norm": 0.4235232030719621,
	"learning_rate": 8.812258202295996e-05,
	"loss": 0.152,
	"step": 3180
	},
	{
	"epoch": 0.2489400599012019,
	"grad_norm": 0.6213092298155933,
	"learning_rate": 8.79590742125752e-05,
	"loss": 0.1546,
	"step": 3200
	},
	{
	"epoch": 0.25049593527558445,
	"grad_norm": 0.47898830484247745,
	"learning_rate": 8.779460254835539e-05,
	"loss": 0.1211,
	"step": 3220
	},
	{
	"epoch": 0.25205181064996696,
	"grad_norm": 0.2954711756068039,
	"learning_rate": 8.76291712065521e-05,
	"loss": 0.1512,
	"step": 3240
	},
	{
	"epoch": 0.25360768602434947,
	"grad_norm": 0.3538759847437944,
	"learning_rate": 8.746278438778504e-05,
	"loss": 0.1513,
	"step": 3260
	},
	{
	"epoch": 0.255163561398732,
	"grad_norm": 0.4113950148493145,
	"learning_rate": 8.729544631693522e-05,
	"loss": 0.1224,
	"step": 3280
	},
	{
	"epoch": 0.2567194367731145,
	"grad_norm": 0.23665428938367375,
	"learning_rate": 8.712716124303782e-05,
	"loss": 0.1211,
	"step": 3300
	},
	{
	"epoch": 0.258275312147497,
	"grad_norm": 0.30894286169724844,
	"learning_rate": 8.695793343917425e-05,
	"loss": 0.1426,
	"step": 3320
	},
	{
	"epoch": 0.2598311875218795,
	"grad_norm": 0.30883178083282137,
	"learning_rate": 8.67877672023636e-05,
	"loss": 0.157,
	"step": 3340
	},
	{
	"epoch": 0.261387062896262,
	"grad_norm": 0.2697995340043806,
	"learning_rate": 8.661666685345364e-05,
	"loss": 0.1414,
	"step": 3360
	},
	{
	"epoch": 0.2629429382706445,
	"grad_norm": 0.34428380505271733,
	"learning_rate": 8.644463673701099e-05,
	"loss": 0.1286,
	"step": 3380
	},
	{
	"epoch": 0.26449881364502703,
	"grad_norm": 0.360050078831324,
	"learning_rate": 8.62716812212109e-05,
	"loss": 0.1343,
	"step": 3400
	},
	{
	"epoch": 0.26605468901940954,
	"grad_norm": 0.2855442803098881,
	"learning_rate": 8.609780469772623e-05,
	"loss": 0.1367,
	"step": 3420
	},
	{
	"epoch": 0.26761056439379205,
	"grad_norm": 0.7297433372512957,
	"learning_rate": 8.592301158161605e-05,
	"loss": 0.1298,
	"step": 3440
	},
	{
	"epoch": 0.26916643976817456,
	"grad_norm": 0.2573404047035545,
	"learning_rate": 8.574730631121342e-05,
	"loss": 0.1369,
	"step": 3460
	},
	{
	"epoch": 0.27072231514255707,
	"grad_norm": 0.46233001632590615,
	"learning_rate": 8.557069334801282e-05,
	"loss": 0.1356,
	"step": 3480
	},
	{
	"epoch": 0.2722781905169396,
	"grad_norm": 0.3093245413947113,
	"learning_rate": 8.539317717655675e-05,
	"loss": 0.1152,
	"step": 3500
	},
	{
	"epoch": 0.2738340658913221,
	"grad_norm": 0.312057716886946,
	"learning_rate": 8.521476230432191e-05,
	"loss": 0.1279,
	"step": 3520
	},
	{
	"epoch": 0.2753899412657046,
	"grad_norm": 0.2290107529755153,
	"learning_rate": 8.503545326160471e-05,
	"loss": 0.1632,
	"step": 3540
	},
	{
	"epoch": 0.2769458166400871,
	"grad_norm": 0.21905580988299717,
	"learning_rate": 8.485525460140636e-05,
	"loss": 0.1418,
	"step": 3560
	},
	{
	"epoch": 0.2785016920144696,
	"grad_norm": 0.3925924615260156,
	"learning_rate": 8.467417089931706e-05,
	"loss": 0.1254,
	"step": 3580
	},
	{
	"epoch": 0.2800575673888522,
	"grad_norm": 0.30442895135731385,
	"learning_rate": 8.44922067534e-05,
	"loss": 0.1169,
	"step": 3600
	},
	{
	"epoch": 0.2816134427632347,
	"grad_norm": 0.49632910550108267,
	"learning_rate": 8.430936678407446e-05,
	"loss": 0.1167,
	"step": 3620
	},
	{
	"epoch": 0.2831693181376172,
	"grad_norm": 0.24623998244883621,
	"learning_rate": 8.412565563399863e-05,
	"loss": 0.1342,
	"step": 3640
	},
	{
	"epoch": 0.2847251935119997,
	"grad_norm": 0.28488079021241974,
	"learning_rate": 8.394107796795164e-05,
	"loss": 0.1242,
	"step": 3660
	},
	{
	"epoch": 0.2862810688863822,
	"grad_norm": 0.31194070778134303,
	"learning_rate": 8.375563847271506e-05,
	"loss": 0.1296,
	"step": 3680
	},
	{
	"epoch": 0.2878369442607647,
	"grad_norm": 0.4160360279208812,
	"learning_rate": 8.356934185695404e-05,
	"loss": 0.1694,
	"step": 3700
	},
	{
	"epoch": 0.28939281963514724,
	"grad_norm": 0.3166176821118176,
	"learning_rate": 8.338219285109765e-05,
	"loss": 0.1506,
	"step": 3720
	},
	{
	"epoch": 0.29094869500952975,
	"grad_norm": 0.32757798146707673,
	"learning_rate": 8.319419620721873e-05,
	"loss": 0.1428,
	"step": 3740
	},
	{
	"epoch": 0.29250457038391225,
	"grad_norm": 0.2602037486009897,
	"learning_rate": 8.300535669891334e-05,
	"loss": 0.1111,
	"step": 3760
	},
	{
	"epoch": 0.29406044575829476,
	"grad_norm": 0.3639880396814407,
	"learning_rate": 8.281567912117944e-05,
	"loss": 0.1284,
	"step": 3780
	},
	{
	"epoch": 0.2956163211326773,
	"grad_norm": 0.24045101314949172,
	"learning_rate": 8.262516829029524e-05,
	"loss": 0.1204,
	"step": 3800
	},
	{
	"epoch": 0.2971721965070598,
	"grad_norm": 0.5932433588553356,
	"learning_rate": 8.243382904369677e-05,
	"loss": 0.1304,
	"step": 3820
	},
	{
	"epoch": 0.2987280718814423,
	"grad_norm": 0.23610914523714283,
	"learning_rate": 8.224166623985522e-05,
	"loss": 0.1355,
	"step": 3840
	},
	{
	"epoch": 0.3002839472558248,
	"grad_norm": 0.26306942447548964,
	"learning_rate": 8.20486847581534e-05,
	"loss": 0.1653,
	"step": 3860
	},
	{
	"epoch": 0.3018398226302073,
	"grad_norm": 0.3837730281977164,
	"learning_rate": 8.185488949876198e-05,
	"loss": 0.1488,
	"step": 3880
	},
	{
	"epoch": 0.3033956980045898,
	"grad_norm": 0.2075932638103982,
	"learning_rate": 8.166028538251497e-05,
	"loss": 0.1186,
	"step": 3900
	},
	{
	"epoch": 0.30495157337897233,
	"grad_norm": 0.7439842401042975,
	"learning_rate": 8.146487735078483e-05,
	"loss": 0.1596,
	"step": 3920
	},
	{
	"epoch": 0.30650744875335484,
	"grad_norm": 0.19563507811342412,
	"learning_rate": 8.126867036535697e-05,
	"loss": 0.0821,
	"step": 3940
	},
	{
	"epoch": 0.30806332412773735,
	"grad_norm": 0.2965603442960315,
	"learning_rate": 8.107166940830377e-05,
	"loss": 0.1517,
	"step": 3960
	},
	{
	"epoch": 0.30961919950211986,
	"grad_norm": 0.3187610302560432,
	"learning_rate": 8.08738794818581e-05,
	"loss": 0.134,
	"step": 3980
	},
	{
	"epoch": 0.31117507487650237,
	"grad_norm": 0.3368367019972521,
	"learning_rate": 8.067530560828626e-05,
	"loss": 0.1168,
	"step": 4000
	},
	{
	"epoch": 0.31273095025088493,
	"grad_norm": 0.387956931911519,
	"learning_rate": 8.047595282976044e-05,
	"loss": 0.1442,
	"step": 4020
	},
	{
	"epoch": 0.31428682562526744,
	"grad_norm": 0.38486448272072704,
	"learning_rate": 8.027582620823078e-05,
	"loss": 0.1587,
	"step": 4040
	},
	{
	"epoch": 0.31584270099964995,
	"grad_norm": 0.5533522908972893,
	"learning_rate": 8.007493082529674e-05,
	"loss": 0.1241,
	"step": 4060
	},
	{
	"epoch": 0.31739857637403246,
	"grad_norm": 2.6650250431553832,
	"learning_rate": 7.987327178207815e-05,
	"loss": 0.1175,
	"step": 4080
	},
	{
	"epoch": 0.31895445174841497,
	"grad_norm": 0.34837618264953285,
	"learning_rate": 7.967085419908563e-05,
	"loss": 0.1319,
	"step": 4100
	},
	{
	"epoch": 0.3205103271227975,
	"grad_norm": 0.23482314150136452,
	"learning_rate": 7.94676832160905e-05,
	"loss": 0.1398,
	"step": 4120
	},
	{
	"epoch": 0.32206620249718,
	"grad_norm": 0.26922948370820765,
	"learning_rate": 7.92637639919945e-05,
	"loss": 0.1146,
	"step": 4140
	},
	{
	"epoch": 0.3236220778715625,
	"grad_norm": 0.23460187372199875,
	"learning_rate": 7.905910170469853e-05,
	"loss": 0.1307,
	"step": 4160
	},
	{
	"epoch": 0.325177953245945,
	"grad_norm": 0.5910478026798753,
	"learning_rate": 7.885370155097135e-05,
	"loss": 0.142,
	"step": 4180
	},
	{
	"epoch": 0.3267338286203275,
	"grad_norm": 0.3610158461245757,
	"learning_rate": 7.864756874631756e-05,
	"loss": 0.1158,
	"step": 4200
	},
	{
	"epoch": 0.32828970399471,
	"grad_norm": 0.35957535703230487,
	"learning_rate": 7.844070852484516e-05,
	"loss": 0.1293,
	"step": 4220
	},
	{
	"epoch": 0.32984557936909253,
	"grad_norm": 0.27293200693843817,
	"learning_rate": 7.823312613913268e-05,
	"loss": 0.1339,
	"step": 4240
	},
	{
	"epoch": 0.33140145474347504,
	"grad_norm": 0.33008985671996344,
	"learning_rate": 7.802482686009576e-05,
	"loss": 0.146,
	"step": 4260
	},
	{
	"epoch": 0.33295733011785755,
	"grad_norm": 0.2631270850466356,
	"learning_rate": 7.781581597685336e-05,
	"loss": 0.1263,
	"step": 4280
	},
	{
	"epoch": 0.33451320549224006,
	"grad_norm": 0.2889987019534576,
	"learning_rate": 7.760609879659341e-05,
	"loss": 0.1435,
	"step": 4300
	},
	{
	"epoch": 0.33606908086662257,
	"grad_norm": 0.4206728920791759,
	"learning_rate": 7.739568064443816e-05,
	"loss": 0.1275,
	"step": 4320
	},
	{
	"epoch": 0.3376249562410051,
	"grad_norm": 0.3773718424449096,
	"learning_rate": 7.718456686330877e-05,
	"loss": 0.1157,
	"step": 4340
	},
	{
	"epoch": 0.3391808316153876,
	"grad_norm": 0.29162040161915054,
	"learning_rate": 7.697276281378984e-05,
	"loss": 0.1312,
	"step": 4360
	},
	{
	"epoch": 0.3407367069897701,
	"grad_norm": 0.2586120415468226,
	"learning_rate": 7.676027387399318e-05,
	"loss": 0.1437,
	"step": 4380
	},
	{
	"epoch": 0.3422925823641526,
	"grad_norm": 0.22036686428611407,
	"learning_rate": 7.654710543942126e-05,
	"loss": 0.1169,
	"step": 4400
	},
	{
	"epoch": 0.3438484577385351,
	"grad_norm": 0.47430371225888784,
	"learning_rate": 7.633326292283028e-05,
	"loss": 0.1207,
	"step": 4420
	},
	{
	"epoch": 0.3454043331129177,
	"grad_norm": 0.5860082160676681,
	"learning_rate": 7.611875175409264e-05,
	"loss": 0.1479,
	"step": 4440
	},
	{
	"epoch": 0.3469602084873002,
	"grad_norm": 0.3150995230171685,
	"learning_rate": 7.590357738005911e-05,
	"loss": 0.1067,
	"step": 4460
	},
	{
	"epoch": 0.3485160838616827,
	"grad_norm": 0.8191848750717672,
	"learning_rate": 7.568774526442051e-05,
	"loss": 0.1168,
	"step": 4480
	},
	{
	"epoch": 0.3500719592360652,
	"grad_norm": 0.30197587725111064,
	"learning_rate": 7.547126088756898e-05,
	"loss": 0.1423,
	"step": 4500
	},
	{
	"epoch": 0.3516278346104477,
	"grad_norm": 0.2554021558180106,
	"learning_rate": 7.525412974645886e-05,
	"loss": 0.1086,
	"step": 4520
	},
	{
	"epoch": 0.35318370998483023,
	"grad_norm": 0.5628930583609504,
	"learning_rate": 7.503635735446708e-05,
	"loss": 0.1158,
	"step": 4540
	},
	{
	"epoch": 0.35473958535921274,
	"grad_norm": 0.3298741038732343,
	"learning_rate": 7.481794924125308e-05,
	"loss": 0.1167,
	"step": 4560
	},
	{
	"epoch": 0.35629546073359525,
	"grad_norm": 0.304803035761189,
	"learning_rate": 7.459891095261856e-05,
	"loss": 0.1183,
	"step": 4580
	},
	{
	"epoch": 0.35785133610797776,
	"grad_norm": 1.106406177544165,
	"learning_rate": 7.437924805036657e-05,
	"loss": 0.1375,
	"step": 4600
	},
	{
	"epoch": 0.35940721148236027,
	"grad_norm": 0.32239423443964793,
	"learning_rate": 7.415896611216029e-05,
	"loss": 0.1287,
	"step": 4620
	},
	{
	"epoch": 0.3609630868567428,
	"grad_norm": 0.32655493002753033,
	"learning_rate": 7.393807073138143e-05,
	"loss": 0.1154,
	"step": 4640
	},
	{
	"epoch": 0.3625189622311253,
	"grad_norm": 0.2651412760181757,
	"learning_rate": 7.371656751698815e-05,
	"loss": 0.1205,
	"step": 4660
	},
	{
	"epoch": 0.3640748376055078,
	"grad_norm": 0.2099783368787053,
	"learning_rate": 7.349446209337272e-05,
	"loss": 0.1277,
	"step": 4680
	},
	{
	"epoch": 0.3656307129798903,
	"grad_norm": 0.18988639504828586,
	"learning_rate": 7.327176010021862e-05,
	"loss": 0.13,
	"step": 4700
	},
	{
	"epoch": 0.3671865883542728,
	"grad_norm": 0.29678365946409435,
	"learning_rate": 7.30484671923574e-05,
	"loss": 0.1322,
	"step": 4720
	},
	{
	"epoch": 0.3687424637286553,
	"grad_norm": 0.2224201871860834,
	"learning_rate": 7.282458903962505e-05,
	"loss": 0.1613,
	"step": 4740
	},
	{
	"epoch": 0.37029833910303783,
	"grad_norm": 0.27548285006924644,
	"learning_rate": 7.260013132671808e-05,
	"loss": 0.1266,
	"step": 4760
	},
	{
	"epoch": 0.37185421447742034,
	"grad_norm": 0.29277553770003534,
	"learning_rate": 7.237509975304912e-05,
	"loss": 0.1364,
	"step": 4780
	},
	{
	"epoch": 0.37341008985180285,
	"grad_norm": 0.3274266777526162,
	"learning_rate": 7.214950003260224e-05,
	"loss": 0.1325,
	"step": 4800
	},
	{
	"epoch": 0.37496596522618536,
	"grad_norm": 0.47790297447775604,
	"learning_rate": 7.192333789378786e-05,
	"loss": 0.1366,
	"step": 4820
	},
	{
	"epoch": 0.37652184060056787,
	"grad_norm": 1.9421839263555873,
	"learning_rate": 7.169661907929724e-05,
	"loss": 0.1275,
	"step": 4840
	},
	{
	"epoch": 0.3780777159749504,
	"grad_norm": 0.4845237474268914,
	"learning_rate": 7.146934934595677e-05,
	"loss": 0.115,
	"step": 4860
	},
	{
	"epoch": 0.37963359134933294,
	"grad_norm": 0.2941830305413283,
	"learning_rate": 7.124153446458165e-05,
	"loss": 0.1478,
	"step": 4880
	},
	{
	"epoch": 0.38118946672371545,
	"grad_norm": 0.24862890190996334,
	"learning_rate": 7.101318021982952e-05,
	"loss": 0.1065,
	"step": 4900
	},
	{
	"epoch": 0.38274534209809796,
	"grad_norm": 0.2696715251135114,
	"learning_rate": 7.078429241005344e-05,
	"loss": 0.1219,
	"step": 4920
	},
	{
	"epoch": 0.38430121747248047,
	"grad_norm": 0.27582537022149595,
	"learning_rate": 7.05548768471547e-05,
	"loss": 0.1559,
	"step": 4940
	},
	{
	"epoch": 0.385857092846863,
	"grad_norm": 0.5205528692136754,
	"learning_rate": 7.032493935643531e-05,
	"loss": 0.1311,
	"step": 4960
	},
	{
	"epoch": 0.3874129682212455,
	"grad_norm": 0.7113565087691933,
	"learning_rate": 7.009448577644999e-05,
	"loss": 0.1261,
	"step": 4980
	},
	{
	"epoch": 0.388968843595628,
	"grad_norm": 0.26153853335362887,
	"learning_rate": 6.986352195885796e-05,
	"loss": 0.1125,
	"step": 5000
	},
	{
	"epoch": 0.3905247189700105,
	"grad_norm": 0.34720126482225333,
	"learning_rate": 6.963205376827437e-05,
	"loss": 0.1442,
	"step": 5020
	},
	{
	"epoch": 0.392080594344393,
	"grad_norm": 0.2845625360912405,
	"learning_rate": 6.940008708212133e-05,
	"loss": 0.1447,
	"step": 5040
	},
	{
	"epoch": 0.39363646971877553,
	"grad_norm": 0.28643713119668573,
	"learning_rate": 6.916762779047878e-05,
	"loss": 0.1357,
	"step": 5060
	},
	{
	"epoch": 0.39519234509315804,
	"grad_norm": 0.41353697729139904,
	"learning_rate": 6.893468179593481e-05,
	"loss": 0.1144,
	"step": 5080
	},
	{
	"epoch": 0.39674822046754055,
	"grad_norm": 0.6100468637182057,
	"learning_rate": 6.870125501343583e-05,
	"loss": 0.1341,
	"step": 5100
	},
	{
	"epoch": 0.39830409584192306,
	"grad_norm": 0.24886968948586063,
	"learning_rate": 6.84673533701364e-05,
	"loss": 0.1142,
	"step": 5120
	},
	{
	"epoch": 0.39985997121630557,
	"grad_norm": 0.2538953251466427,
	"learning_rate": 6.823298280524868e-05,
	"loss": 0.1147,
	"step": 5140
	},
	{
	"epoch": 0.4014158465906881,
	"grad_norm": 0.2005574520325003,
	"learning_rate": 6.79981492698917e-05,
	"loss": 0.1232,
	"step": 5160
	},
	{
	"epoch": 0.4029717219650706,
	"grad_norm": 0.5221937220028142,
	"learning_rate": 6.776285872694018e-05,
	"loss": 0.1402,
	"step": 5180
	},
	{
	"epoch": 0.4045275973394531,
	"grad_norm": 0.207680805552986,
	"learning_rate": 6.752711715087309e-05,
	"loss": 0.1324,
	"step": 5200
	},
	{
	"epoch": 0.4060834727138356,
	"grad_norm": 0.30078856208086296,
	"learning_rate": 6.729093052762207e-05,
	"loss": 0.1271,
	"step": 5220
	},
	{
	"epoch": 0.4076393480882181,
	"grad_norm": 0.20287209442925436,
	"learning_rate": 6.705430485441931e-05,
	"loss": 0.1256,
	"step": 5240
	},
	{
	"epoch": 0.4091952234626006,
	"grad_norm": 0.23520832779073356,
	"learning_rate": 6.681724613964536e-05,
	"loss": 0.1162,
	"step": 5260
	},
	{
	"epoch": 0.41075109883698313,
	"grad_norm": 0.24118148677817738,
	"learning_rate": 6.657976040267652e-05,
	"loss": 0.1196,
	"step": 5280
	},
	{
	"epoch": 0.4123069742113657,
	"grad_norm": 0.1838128494029971,
	"learning_rate": 6.634185367373197e-05,
	"loss": 0.1248,
	"step": 5300
	},
	{
	"epoch": 0.4138628495857482,
	"grad_norm": 0.2719129142489373,
	"learning_rate": 6.610353199372072e-05,
	"loss": 0.127,
	"step": 5320
	},
	{
	"epoch": 0.4154187249601307,
	"grad_norm": 0.23580716485035527,
	"learning_rate": 6.58648014140882e-05,
	"loss": 0.1138,
	"step": 5340
	},
	{
	"epoch": 0.4169746003345132,
	"grad_norm": 0.3287090754743541,
	"learning_rate": 6.562566799666254e-05,
	"loss": 0.1223,
	"step": 5360
	},
	{
	"epoch": 0.41853047570889573,
	"grad_norm": 0.19932618816745395,
	"learning_rate": 6.538613781350071e-05,
	"loss": 0.1094,
	"step": 5380
	},
	{
	"epoch": 0.42008635108327824,
	"grad_norm": 0.22444078324072805,
	"learning_rate": 6.514621694673434e-05,
	"loss": 0.1144,
	"step": 5400
	},
	{
	"epoch": 0.42164222645766075,
	"grad_norm": 0.1655744100969424,
	"learning_rate": 6.490591148841527e-05,
	"loss": 0.1148,
	"step": 5420
	},
	{
	"epoch": 0.42319810183204326,
	"grad_norm": 0.28735657943351894,
	"learning_rate": 6.466522754036085e-05,
	"loss": 0.1304,
	"step": 5440
	},
	{
	"epoch": 0.42475397720642577,
	"grad_norm": 0.2185687233092101,
	"learning_rate": 6.442417121399901e-05,
	"loss": 0.1475,
	"step": 5460
	},
	{
	"epoch": 0.4263098525808083,
	"grad_norm": 0.48107992164889046,
	"learning_rate": 6.418274863021307e-05,
	"loss": 0.1165,
	"step": 5480
	},
	{
	"epoch": 0.4278657279551908,
	"grad_norm": 0.15979990906342145,
	"learning_rate": 6.394096591918635e-05,
	"loss": 0.1176,
	"step": 5500
	},
	{
	"epoch": 0.4294216033295733,
	"grad_norm": 0.8071106785141181,
	"learning_rate": 6.369882922024648e-05,
	"loss": 0.1382,
	"step": 5520
	},
	{
	"epoch": 0.4309774787039558,
	"grad_norm": 0.3221087516079968,
	"learning_rate": 6.345634468170954e-05,
	"loss": 0.104,
	"step": 5540
	},
	{
	"epoch": 0.4325333540783383,
	"grad_norm": 0.295937302225145,
	"learning_rate": 6.321351846072388e-05,
	"loss": 0.1404,
	"step": 5560
	},
	{
	"epoch": 0.4340892294527208,
	"grad_norm": 0.25799115943610434,
	"learning_rate": 6.297035672311389e-05,
	"loss": 0.1114,
	"step": 5580
	},
	{
	"epoch": 0.43564510482710334,
	"grad_norm": 0.3111200136613854,
	"learning_rate": 6.272686564322333e-05,
	"loss": 0.1396,
	"step": 5600
	},
	{
	"epoch": 0.43720098020148584,
	"grad_norm": 0.38997955372510623,
	"learning_rate": 6.248305140375861e-05,
	"loss": 0.1407,
	"step": 5620
	},
	{
	"epoch": 0.43875685557586835,
	"grad_norm": 0.28005150715813815,
	"learning_rate": 6.223892019563174e-05,
	"loss": 0.1228,
	"step": 5640
	},
	{
	"epoch": 0.44031273095025086,
	"grad_norm": 0.23708090475106436,
	"learning_rate": 6.199447821780325e-05,
	"loss": 0.1149,
	"step": 5660
	},
	{
	"epoch": 0.4418686063246334,
	"grad_norm": 0.2971036099131731,
	"learning_rate": 6.174973167712464e-05,
	"loss": 0.1195,
	"step": 5680
	},
	{
	"epoch": 0.4434244816990159,
	"grad_norm": 0.33863438698786524,
	"learning_rate": 6.15046867881809e-05,
	"loss": 0.1223,
	"step": 5700
	},
	{
	"epoch": 0.44498035707339845,
	"grad_norm": 0.2779208880093748,
	"learning_rate": 6.125934977313261e-05,
	"loss": 0.0866,
	"step": 5720
	},
	{
	"epoch": 0.44653623244778096,
	"grad_norm": 0.22800068088428616,
	"learning_rate": 6.101372686155802e-05,
	"loss": 0.1342,
	"step": 5740
	},
	{
	"epoch": 0.44809210782216347,
	"grad_norm": 0.20706392154332443,
	"learning_rate": 6.076782429029483e-05,
	"loss": 0.1346,
	"step": 5760
	},
	{
	"epoch": 0.449647983196546,
	"grad_norm": 0.33407841109179065,
	"learning_rate": 6.052164830328184e-05,
	"loss": 0.1003,
	"step": 5780
	},
	{
	"epoch": 0.4512038585709285,
	"grad_norm": 0.3016367709989675,
	"learning_rate": 6.027520515140041e-05,
	"loss": 0.1103,
	"step": 5800
	},
	{
	"epoch": 0.452759733945311,
	"grad_norm": 0.1578695554788324,
	"learning_rate": 6.002850109231573e-05,
	"loss": 0.1015,
	"step": 5820
	},
	{
	"epoch": 0.4543156093196935,
	"grad_norm": 0.21991187594278833,
	"learning_rate": 5.978154239031789e-05,
	"loss": 0.139,
	"step": 5840
	},
	{
	"epoch": 0.455871484694076,
	"grad_norm": 0.2874696210148598,
	"learning_rate": 5.9534335316162914e-05,
	"loss": 0.123,
	"step": 5860
	},
	{
	"epoch": 0.4574273600684585,
	"grad_norm": 0.2772586670528982,
	"learning_rate": 5.9286886146913424e-05,
	"loss": 0.1154,
	"step": 5880
	},
	{
	"epoch": 0.45898323544284103,
	"grad_norm": 0.37689072367994886,
	"learning_rate": 5.9039201165779315e-05,
	"loss": 0.1345,
	"step": 5900
	},
	{
	"epoch": 0.46053911081722354,
	"grad_norm": 0.31557736706996786,
	"learning_rate": 5.879128666195819e-05,
	"loss": 0.1253,
	"step": 5920
	},
	{
	"epoch": 0.46209498619160605,
	"grad_norm": 0.2634040987367494,
	"learning_rate": 5.85431489304757e-05,
	"loss": 0.1017,
	"step": 5940
	},
	{
	"epoch": 0.46365086156598856,
	"grad_norm": 0.5745291942146606,
	"learning_rate": 5.829479427202567e-05,
	"loss": 0.1329,
	"step": 5960
	},
	{
	"epoch": 0.46520673694037107,
	"grad_norm": 0.2768443524567914,
	"learning_rate": 5.804622899281008e-05,
	"loss": 0.1082,
	"step": 5980
	},
	{
	"epoch": 0.4667626123147536,
	"grad_norm": 0.2642503210264253,
	"learning_rate": 5.779745940437903e-05,
	"loss": 0.1235,
	"step": 6000
	},
	{
	"epoch": 0.4683184876891361,
	"grad_norm": 0.3292385903992304,
	"learning_rate": 5.7548491823470376e-05,
	"loss": 0.1224,
	"step": 6020
	},
	{
	"epoch": 0.4698743630635186,
	"grad_norm": 0.20779016305278444,
	"learning_rate": 5.729933257184943e-05,
	"loss": 0.1014,
	"step": 6040
	},
	{
	"epoch": 0.4714302384379011,
	"grad_norm": 0.4190445082479885,
	"learning_rate": 5.704998797614832e-05,
	"loss": 0.1124,
	"step": 6060
	},
	{
	"epoch": 0.4729861138122836,
	"grad_norm": 0.8100001222940857,
	"learning_rate": 5.680046436770551e-05,
	"loss": 0.1048,
	"step": 6080
	},
	{
	"epoch": 0.4745419891866661,
	"grad_norm": 0.32810084977296833,
	"learning_rate": 5.655076808240487e-05,
	"loss": 0.1154,
	"step": 6100
	},
	{
	"epoch": 0.47609786456104863,
	"grad_norm": 0.31865452063327415,
	"learning_rate": 5.6300905460514895e-05,
	"loss": 0.1356,
	"step": 6120
	},
	{
	"epoch": 0.4776537399354312,
	"grad_norm": 0.3826899711021525,
	"learning_rate": 5.6050882846527695e-05,
	"loss": 0.1324,
	"step": 6140
	},
	{
	"epoch": 0.4792096153098137,
	"grad_norm": 0.29805169842712154,
	"learning_rate": 5.580070658899785e-05,
	"loss": 0.1105,
	"step": 6160
	},
	{
	"epoch": 0.4807654906841962,
	"grad_norm": 0.24977374740124644,
	"learning_rate": 5.5550383040381296e-05,
	"loss": 0.1198,
	"step": 6180
	},
	{
	"epoch": 0.4823213660585787,
	"grad_norm": 0.22038334221301611,
	"learning_rate": 5.529991855687391e-05,
	"loss": 0.0912,
	"step": 6200
	},
	{
	"epoch": 0.48387724143296124,
	"grad_norm": 0.2817216779666876,
	"learning_rate": 5.504931949825024e-05,
	"loss": 0.1339,
	"step": 6220
	},
	{
	"epoch": 0.48543311680734375,
	"grad_norm": 0.26380564128224937,
	"learning_rate": 5.4798592227701917e-05,
	"loss": 0.1216,
	"step": 6240
	},
	{
	"epoch": 0.48698899218172625,
	"grad_norm": 0.2904024951361601,
	"learning_rate": 5.454774311167614e-05,
	"loss": 0.1393,
	"step": 6260
	},
	{
	"epoch": 0.48854486755610876,
	"grad_norm": 0.298657193170181,
	"learning_rate": 5.4296778519713976e-05,
	"loss": 0.1079,
	"step": 6280
	},
	{
	"epoch": 0.4901007429304913,
	"grad_norm": 0.23431996717663298,
	"learning_rate": 5.404570482428866e-05,
	"loss": 0.1307,
	"step": 6300
	},
	{
	"epoch": 0.4916566183048738,
	"grad_norm": 0.25155480876194364,
	"learning_rate": 5.379452840064376e-05,
	"loss": 0.1129,
	"step": 6320
	},
	{
	"epoch": 0.4932124936792563,
	"grad_norm": 0.27491954341811353,
	"learning_rate": 5.354325562663135e-05,
	"loss": 0.1031,
	"step": 6340
	},
	{
	"epoch": 0.4947683690536388,
	"grad_norm": 0.1719629204475533,
	"learning_rate": 5.329189288254999e-05,
	"loss": 0.1075,
	"step": 6360
	},
	{
	"epoch": 0.4963242444280213,
	"grad_norm": 0.2545907189120072,
	"learning_rate": 5.304044655098275e-05,
	"loss": 0.1284,
	"step": 6380
	},
	{
	"epoch": 0.4978801198024038,
	"grad_norm": 0.38294456555599227,
	"learning_rate": 5.278892301663517e-05,
	"loss": 0.1105,
	"step": 6400
	},
	{
	"epoch": 0.49943599517678633,
	"grad_norm": 0.32546741357325526,
	"learning_rate": 5.25373286661731e-05,
	"loss": 0.1125,
	"step": 6420
	},
	{
	"epoch": 0.5009918705511689,
	"grad_norm": 0.28071911797088905,
	"learning_rate": 5.228566988806056e-05,
	"loss": 0.0921,
	"step": 6440
	},
	{
	"epoch": 0.5025477459255514,
	"grad_norm": 0.29859135258429775,
	"learning_rate": 5.20339530723975e-05,
	"loss": 0.1202,
	"step": 6460
	},
	{
	"epoch": 0.5041036212999339,
	"grad_norm": 0.26636000821346184,
	"learning_rate": 5.1782184610757555e-05,
	"loss": 0.0925,
	"step": 6480
	},
	{
	"epoch": 0.5056594966743164,
	"grad_norm": 0.26867586163403573,
	"learning_rate": 5.153037089602577e-05,
	"loss": 0.1033,
	"step": 6500
	},
	{
	"epoch": 0.5072153720486989,
	"grad_norm": 0.1970319353533795,
	"learning_rate": 5.127851832223623e-05,
	"loss": 0.1081,
	"step": 6520
	},
	{
	"epoch": 0.5087712474230814,
	"grad_norm": 0.16862934400112387,
	"learning_rate": 5.1026633284409744e-05,
	"loss": 0.1198,
	"step": 6540
	},
	{
	"epoch": 0.510327122797464,
	"grad_norm": 0.2070616421963743,
	"learning_rate": 5.077472217839141e-05,
	"loss": 0.1384,
	"step": 6560
	},
	{
	"epoch": 0.5118829981718465,
	"grad_norm": 0.23459240983018564,
	"learning_rate": 5.0522791400688284e-05,
	"loss": 0.1167,
	"step": 6580
	},
	{
	"epoch": 0.513438873546229,
	"grad_norm": 0.14869574606175753,
	"learning_rate": 5.0270847348306916e-05,
	"loss": 0.1187,
	"step": 6600
	},
	{
	"epoch": 0.5149947489206115,
	"grad_norm": 0.2711469583506501,
	"learning_rate": 5.0018896418590935e-05,
	"loss": 0.1182,
	"step": 6620
	},
	{
	"epoch": 0.516550624294994,
	"grad_norm": 0.14841248380962843,
	"learning_rate": 4.976694500905857e-05,
	"loss": 0.0949,
	"step": 6640
	},
	{
	"epoch": 0.5181064996693765,
	"grad_norm": 0.19517217863232764,
	"learning_rate": 4.951499951724026e-05,
	"loss": 0.1221,
	"step": 6660
	},
	{
	"epoch": 0.519662375043759,
	"grad_norm": 0.2648426124923396,
	"learning_rate": 4.926306634051616e-05,
	"loss": 0.1222,
	"step": 6680
	},
	{
	"epoch": 0.5212182504181415,
	"grad_norm": 0.3751790940385581,
	"learning_rate": 4.901115187595375e-05,
	"loss": 0.1439,
	"step": 6700
	},
	{
	"epoch": 0.522774125792524,
	"grad_norm": 0.2483857089503091,
	"learning_rate": 4.875926252014534e-05,
	"loss": 0.0994,
	"step": 6720
	},
	{
	"epoch": 0.5243300011669065,
	"grad_norm": 0.3144728908519791,
	"learning_rate": 4.8507404669045684e-05,
	"loss": 0.1259,
	"step": 6740
	},
	{
	"epoch": 0.525885876541289,
	"grad_norm": 0.36980806319362897,
	"learning_rate": 4.825558471780961e-05,
	"loss": 0.1112,
	"step": 6760
	},
	{
	"epoch": 0.5274417519156716,
	"grad_norm": 0.22295154371115505,
	"learning_rate": 4.800380906062956e-05,
	"loss": 0.1172,
	"step": 6780
	},
	{
	"epoch": 0.5289976272900541,
	"grad_norm": 0.27974515454024806,
	"learning_rate": 4.775208409057327e-05,
	"loss": 0.0995,
	"step": 6800
	},
	{
	"epoch": 0.5305535026644366,
	"grad_norm": 0.3918269231188076,
	"learning_rate": 4.750041619942143e-05,
	"loss": 0.1232,
	"step": 6820
	},
	{
	"epoch": 0.5321093780388191,
	"grad_norm": 0.3489507740890769,
	"learning_rate": 4.7248811777505404e-05,
	"loss": 0.1044,
	"step": 6840
	},
	{
	"epoch": 0.5336652534132016,
	"grad_norm": 0.20080963454825448,
	"learning_rate": 4.699727721354493e-05,
	"loss": 0.1432,
	"step": 6860
	},
	{
	"epoch": 0.5352211287875841,
	"grad_norm": 0.2563329338228848,
	"learning_rate": 4.674581889448591e-05,
	"loss": 0.1114,
	"step": 6880
	},
	{
	"epoch": 0.5367770041619666,
	"grad_norm": 0.29485801432989583,
	"learning_rate": 4.64944432053383e-05,
	"loss": 0.1356,
	"step": 6900
	},
	{
	"epoch": 0.5383328795363491,
	"grad_norm": 0.2552971805047558,
	"learning_rate": 4.6243156529013876e-05,
	"loss": 0.1279,
	"step": 6920
	},
	{
	"epoch": 0.5398887549107316,
	"grad_norm": 0.20589417883812308,
	"learning_rate": 4.5991965246164173e-05,
	"loss": 0.1308,
	"step": 6940
	},
	{
	"epoch": 0.5414446302851141,
	"grad_norm": 0.24236643762005394,
	"learning_rate": 4.574087573501855e-05,
	"loss": 0.1272,
	"step": 6960
	},
	{
	"epoch": 0.5430005056594966,
	"grad_norm": 0.37493382197055836,
	"learning_rate": 4.5489894371222214e-05,
	"loss": 0.1167,
	"step": 6980
	},
	{
	"epoch": 0.5445563810338792,
	"grad_norm": 0.26497900213884285,
	"learning_rate": 4.5239027527674244e-05,
	"loss": 0.1137,
	"step": 7000
	},
	{
	"epoch": 0.5461122564082617,
	"grad_norm": 1.1616125886640192,
	"learning_rate": 4.4988281574365864e-05,
	"loss": 0.0982,
	"step": 7020
	},
	{
	"epoch": 0.5476681317826442,
	"grad_norm": 0.20125501810192353,
	"learning_rate": 4.473766287821865e-05,
	"loss": 0.1051,
	"step": 7040
	},
	{
	"epoch": 0.5492240071570267,
	"grad_norm": 0.22878295385327083,
	"learning_rate": 4.4487177802922867e-05,
	"loss": 0.12,
	"step": 7060
	},
	{
	"epoch": 0.5507798825314092,
	"grad_norm": 0.31367745310271644,
	"learning_rate": 4.4236832708775924e-05,
	"loss": 0.1137,
	"step": 7080
	},
	{
	"epoch": 0.5523357579057917,
	"grad_norm": 0.1941191241265075,
	"learning_rate": 4.398663395252079e-05,
	"loss": 0.1193,
	"step": 7100
	},
	{
	"epoch": 0.5538916332801742,
	"grad_norm": 0.2543371070937438,
	"learning_rate": 4.373658788718466e-05,
	"loss": 0.105,
	"step": 7120
	},
	{
	"epoch": 0.5554475086545567,
	"grad_norm": 0.39030509368883254,
	"learning_rate": 4.348670086191763e-05,
	"loss": 0.1176,
	"step": 7140
	},
	{
	"epoch": 0.5570033840289392,
	"grad_norm": 0.24085111662300612,
	"learning_rate": 4.3236979221831446e-05,
	"loss": 0.109,
	"step": 7160
	},
	{
	"epoch": 0.5585592594033217,
	"grad_norm": 0.36029559771795855,
	"learning_rate": 4.29874293078384e-05,
	"loss": 0.1003,
	"step": 7180
	},
	{
	"epoch": 0.5601151347777044,
	"grad_norm": 0.20988689655359558,
	"learning_rate": 4.273805745649032e-05,
	"loss": 0.1074,
	"step": 7200
	},
	{
	"epoch": 0.5616710101520869,
	"grad_norm": 0.2262912277381537,
	"learning_rate": 4.2488869999817715e-05,
	"loss": 0.1432,
	"step": 7220
	},
	{
	"epoch": 0.5632268855264694,
	"grad_norm": 0.20242234996343939,
	"learning_rate": 4.223987326516893e-05,
	"loss": 0.116,
	"step": 7240
	},
	{
	"epoch": 0.5647827609008519,
	"grad_norm": 0.3280411479992993,
	"learning_rate": 4.1991073575049514e-05,
	"loss": 0.1153,
	"step": 7260
	},
	{
	"epoch": 0.5663386362752344,
	"grad_norm": 0.12187792800927902,
	"learning_rate": 4.174247724696171e-05,
	"loss": 0.0876,
	"step": 7280
	},
	{
	"epoch": 0.5678945116496169,
	"grad_norm": 0.22524371042162789,
	"learning_rate": 4.149409059324396e-05,
	"loss": 0.1117,
	"step": 7300
	},
	{
	"epoch": 0.5694503870239994,
	"grad_norm": 0.22650032689059016,
	"learning_rate": 4.1245919920910696e-05,
	"loss": 0.1125,
	"step": 7320
	},
	{
	"epoch": 0.5710062623983819,
	"grad_norm": 0.17488752163399976,
	"learning_rate": 4.099797153149217e-05,
	"loss": 0.1004,
	"step": 7340
	},
	{
	"epoch": 0.5725621377727644,
	"grad_norm": 0.19287797517486263,
	"learning_rate": 4.0750251720874425e-05,
	"loss": 0.1408,
	"step": 7360
	},
	{
	"epoch": 0.5741180131471469,
	"grad_norm": 0.2347460743018896,
	"learning_rate": 4.0502766779139484e-05,
	"loss": 0.0932,
	"step": 7380
	},
	{
	"epoch": 0.5756738885215295,
	"grad_norm": 0.2205472225153125,
	"learning_rate": 4.025552299040555e-05,
	"loss": 0.136,
	"step": 7400
	},
	{
	"epoch": 0.577229763895912,
	"grad_norm": 0.6515706400252066,
	"learning_rate": 4.000852663266753e-05,
	"loss": 0.0881,
	"step": 7420
	},
	{
	"epoch": 0.5787856392702945,
	"grad_norm": 0.26603191027113887,
	"learning_rate": 3.976178397763759e-05,
	"loss": 0.0955,
	"step": 7440
	},
	{
	"epoch": 0.580341514644677,
	"grad_norm": 0.1734412870305003,
	"learning_rate": 3.951530129058583e-05,
	"loss": 0.1356,
	"step": 7460
	},
	{
	"epoch": 0.5818973900190595,
	"grad_norm": 0.3550098623218825,
	"learning_rate": 3.926908483018131e-05,
	"loss": 0.1276,
	"step": 7480
	},
	{
	"epoch": 0.583453265393442,
	"grad_norm": 0.15278484213655477,
	"learning_rate": 3.902314084833309e-05,
	"loss": 0.0956,
	"step": 7500
	},
	{
	"epoch": 0.5850091407678245,
	"grad_norm": 0.17030609786533007,
	"learning_rate": 3.877747559003148e-05,
	"loss": 0.1146,
	"step": 7520
	},
	{
	"epoch": 0.586565016142207,
	"grad_norm": 0.29012832358228163,
	"learning_rate": 3.853209529318944e-05,
	"loss": 0.0934,
	"step": 7540
	},
	{
	"epoch": 0.5881208915165895,
	"grad_norm": 0.25774061459395303,
	"learning_rate": 3.828700618848422e-05,
	"loss": 0.1338,
	"step": 7560
	},
	{
	"epoch": 0.589676766890972,
	"grad_norm": 0.2162034328432579,
	"learning_rate": 3.804221449919914e-05,
	"loss": 0.1118,
	"step": 7580
	},
	{
	"epoch": 0.5912326422653545,
	"grad_norm": 0.23184251750105342,
	"learning_rate": 3.779772644106558e-05,
	"loss": 0.111,
	"step": 7600
	},
	{
	"epoch": 0.5927885176397371,
	"grad_norm": 0.26904595745671644,
	"learning_rate": 3.755354822210513e-05,
	"loss": 0.111,
	"step": 7620
	},
	{
	"epoch": 0.5943443930141196,
	"grad_norm": 0.2859926756568586,
	"learning_rate": 3.730968604247197e-05,
	"loss": 0.1083,
	"step": 7640
	},
	{
	"epoch": 0.5959002683885021,
	"grad_norm": 0.169176167047518,
	"learning_rate": 3.706614609429545e-05,
	"loss": 0.1163,
	"step": 7660
	},
	{
	"epoch": 0.5974561437628846,
	"grad_norm": 0.17859963164614123,
	"learning_rate": 3.682293456152283e-05,
	"loss": 0.0981,
	"step": 7680
	},
	{
	"epoch": 0.5990120191372671,
	"grad_norm": 0.2797164847937639,
	"learning_rate": 3.658005761976226e-05,
	"loss": 0.0914,
	"step": 7700
	},
	{
	"epoch": 0.6005678945116496,
	"grad_norm": 0.3233312778854719,
	"learning_rate": 3.633752143612601e-05,
	"loss": 0.1163,
	"step": 7720
	},
	{
	"epoch": 0.6021237698860321,
	"grad_norm": 0.24824925498921302,
	"learning_rate": 3.609533216907383e-05,
	"loss": 0.1045,
	"step": 7740
	},
	{
	"epoch": 0.6036796452604146,
	"grad_norm": 0.2376999997373067,
	"learning_rate": 3.585349596825658e-05,
	"loss": 0.0758,
	"step": 7760
	},
	{
	"epoch": 0.6052355206347971,
	"grad_norm": 0.35057716837526287,
	"learning_rate": 3.56120189743601e-05,
	"loss": 0.1042,
	"step": 7780
	},
	{
	"epoch": 0.6067913960091796,
	"grad_norm": 0.299999191907227,
	"learning_rate": 3.53709073189493e-05,
	"loss": 0.1368,
	"step": 7800
	},
	{
	"epoch": 0.6083472713835621,
	"grad_norm": 0.2777330286761849,
	"learning_rate": 3.513016712431241e-05,
	"loss": 0.1258,
	"step": 7820
	},
	{
	"epoch": 0.6099031467579447,
	"grad_norm": 0.28419056634526896,
	"learning_rate": 3.488980450330558e-05,
	"loss": 0.0933,
	"step": 7840
	},
	{
	"epoch": 0.6114590221323272,
	"grad_norm": 0.37562295700134135,
	"learning_rate": 3.4649825559197616e-05,
	"loss": 0.117,
	"step": 7860
	},
	{
	"epoch": 0.6130148975067097,
	"grad_norm": 0.25371995043690515,
	"learning_rate": 3.441023638551505e-05,
	"loss": 0.09,
	"step": 7880
	},
	{
	"epoch": 0.6145707728810922,
	"grad_norm": 0.29840213197097154,
	"learning_rate": 3.417104306588738e-05,
	"loss": 0.1183,
	"step": 7900
	},
	{
	"epoch": 0.6161266482554747,
	"grad_norm": 0.16094058624645483,
	"learning_rate": 3.393225167389262e-05,
	"loss": 0.1209,
	"step": 7920
	},
	{
	"epoch": 0.6176825236298572,
	"grad_norm": 0.20761366625535688,
	"learning_rate": 3.369386827290309e-05,
	"loss": 0.0897,
	"step": 7940
	},
	{
	"epoch": 0.6192383990042397,
	"grad_norm": 0.17633413664555408,
	"learning_rate": 3.3455898915931373e-05,
	"loss": 0.0753,
	"step": 7960
	},
	{
	"epoch": 0.6207942743786222,
	"grad_norm": 0.12769973417090547,
	"learning_rate": 3.321834964547673e-05,
	"loss": 0.0939,
	"step": 7980
	},
	{
	"epoch": 0.6223501497530047,
	"grad_norm": 0.2086622755865803,
	"learning_rate": 3.2981226493371606e-05,
	"loss": 0.1181,
	"step": 8000
	},
	{
	"epoch": 0.6239060251273872,
	"grad_norm": 0.2518447983987371,
	"learning_rate": 3.274453548062848e-05,
	"loss": 0.1018,
	"step": 8020
	},
	{
	"epoch": 0.6254619005017699,
	"grad_norm": 0.2999833773571733,
	"learning_rate": 3.2508282617286976e-05,
	"loss": 0.1226,
	"step": 8040
	},
	{
	"epoch": 0.6270177758761524,
	"grad_norm": 0.3280693710724173,
	"learning_rate": 3.2272473902261284e-05,
	"loss": 0.0961,
	"step": 8060
	},
	{
	"epoch": 0.6285736512505349,
	"grad_norm": 0.16241178160205144,
	"learning_rate": 3.203711532318779e-05,
	"loss": 0.0829,
	"step": 8080
	},
	{
	"epoch": 0.6301295266249174,
	"grad_norm": 0.18348846754067977,
	"learning_rate": 3.180221285627308e-05,
	"loss": 0.1123,
	"step": 8100
	},
	{
	"epoch": 0.6316854019992999,
	"grad_norm": 0.22596459179009176,
	"learning_rate": 3.1567772466142156e-05,
	"loss": 0.0941,
	"step": 8120
	},
	{
	"epoch": 0.6332412773736824,
	"grad_norm": 0.2993776046334526,
	"learning_rate": 3.1333800105687026e-05,
	"loss": 0.1022,
	"step": 8140
	},
	{
	"epoch": 0.6347971527480649,
	"grad_norm": 0.44308013806256635,
	"learning_rate": 3.11003017159155e-05,
	"loss": 0.0902,
	"step": 8160
	},
	{
	"epoch": 0.6363530281224474,
	"grad_norm": 0.41772418721111065,
	"learning_rate": 3.086728322580041e-05,
	"loss": 0.1049,
	"step": 8180
	},
	{
	"epoch": 0.6379089034968299,
	"grad_norm": 0.15538841062648698,
	"learning_rate": 3.063475055212896e-05,
	"loss": 0.0719,
	"step": 8200
	},
	{
	"epoch": 0.6394647788712124,
	"grad_norm": 0.3112417738505776,
	"learning_rate": 3.0402709599352542e-05,
	"loss": 0.0962,
	"step": 8220
	},
	{
	"epoch": 0.641020654245595,
	"grad_norm": 0.28389135641643526,
	"learning_rate": 3.0171166259436854e-05,
	"loss": 0.1137,
	"step": 8240
	},
	{
	"epoch": 0.6425765296199775,
	"grad_norm": 0.3327708701322423,
	"learning_rate": 2.9940126411712222e-05,
	"loss": 0.1119,
	"step": 8260
	},
	{
	"epoch": 0.64413240499436,
	"grad_norm": 0.26302260001614,
	"learning_rate": 2.970959592272432e-05,
	"loss": 0.1193,
	"step": 8280
	},
	{
	"epoch": 0.6456882803687425,
	"grad_norm": 0.1724827339622872,
	"learning_rate": 2.947958064608526e-05,
	"loss": 0.1094,
	"step": 8300
	},
	{
	"epoch": 0.647244155743125,
	"grad_norm": 0.16846844897096566,
	"learning_rate": 2.9250086422324886e-05,
	"loss": 0.1302,
	"step": 8320
	},
	{
	"epoch": 0.6488000311175075,
	"grad_norm": 0.2570257038056601,
	"learning_rate": 2.9021119078742552e-05,
	"loss": 0.0948,
	"step": 8340
	},
	{
	"epoch": 0.65035590649189,
	"grad_norm": 0.4986309232385393,
	"learning_rate": 2.879268442925907e-05,
	"loss": 0.094,
	"step": 8360
	},
	{
	"epoch": 0.6519117818662725,
	"grad_norm": 0.14217961054117229,
	"learning_rate": 2.8564788274269128e-05,
	"loss": 0.0961,
	"step": 8380
	},
	{
	"epoch": 0.653467657240655,
	"grad_norm": 0.11586238787120823,
	"learning_rate": 2.8337436400494043e-05,
	"loss": 0.1138,
	"step": 8400
	},
	{
	"epoch": 0.6550235326150375,
	"grad_norm": 0.2064054343841982,
	"learning_rate": 2.8110634580834772e-05,
	"loss": 0.0942,
	"step": 8420
	},
	{
	"epoch": 0.65657940798942,
	"grad_norm": 0.23585218637650507,
	"learning_rate": 2.7884388574225316e-05,
	"loss": 0.1018,
	"step": 8440
	},
	{
	"epoch": 0.6581352833638026,
	"grad_norm": 0.20358146155278237,
	"learning_rate": 2.765870412548652e-05,
	"loss": 0.095,
	"step": 8460
	},
	{
	"epoch": 0.6596911587381851,
	"grad_norm": 0.2804178146576407,
	"learning_rate": 2.7433586965180202e-05,
	"loss": 0.0996,
	"step": 8480
	},
	{
	"epoch": 0.6612470341125676,
	"grad_norm": 0.16134168356237577,
	"learning_rate": 2.7209042809463646e-05,
	"loss": 0.0746,
	"step": 8500
	},
	{
	"epoch": 0.6628029094869501,
	"grad_norm": 0.28160624485946517,
	"learning_rate": 2.6985077359944456e-05,
	"loss": 0.1034,
	"step": 8520
	},
	{
	"epoch": 0.6643587848613326,
	"grad_norm": 0.36391602399667494,
	"learning_rate": 2.676169630353573e-05,
	"loss": 0.1181,
	"step": 8540
	},
	{
	"epoch": 0.6659146602357151,
	"grad_norm": 0.270939284033366,
	"learning_rate": 2.6538905312311755e-05,
	"loss": 0.095,
	"step": 8560
	},
	{
	"epoch": 0.6674705356100976,
	"grad_norm": 0.31727292812899927,
	"learning_rate": 2.63167100433639e-05,
	"loss": 0.1232,
	"step": 8580
	},
	{
	"epoch": 0.6690264109844801,
	"grad_norm": 0.18745124679378602,
	"learning_rate": 2.6095116138657004e-05,
	"loss": 0.0988,
	"step": 8600
	},
	{
	"epoch": 0.6705822863588626,
	"grad_norm": 0.2919236120846842,
	"learning_rate": 2.5874129224886095e-05,
	"loss": 0.1075,
	"step": 8620
	},
	{
	"epoch": 0.6721381617332451,
	"grad_norm": 0.1894708365448588,
	"learning_rate": 2.5653754913333593e-05,
	"loss": 0.0863,
	"step": 8640
	},
	{
	"epoch": 0.6736940371076277,
	"grad_norm": 0.28642669144503863,
	"learning_rate": 2.543399879972671e-05,
	"loss": 0.0861,
	"step": 8660
	},
	{
	"epoch": 0.6752499124820102,
	"grad_norm": 0.5192714167917676,
	"learning_rate": 2.521486646409545e-05,
	"loss": 0.1007,
	"step": 8680
	},
	{
	"epoch": 0.6768057878563927,
	"grad_norm": 0.24569389818629123,
	"learning_rate": 2.499636347063089e-05,
	"loss": 0.1212,
	"step": 8700
	},
	{
	"epoch": 0.6783616632307752,
	"grad_norm": 0.21421123491744254,
	"learning_rate": 2.4778495367543903e-05,
	"loss": 0.0933,
	"step": 8720
	},
	{
	"epoch": 0.6799175386051577,
	"grad_norm": 0.3274893130646572,
	"learning_rate": 2.4561267686924267e-05,
	"loss": 0.1057,
	"step": 8740
	},
	{
	"epoch": 0.6814734139795402,
	"grad_norm": 0.26251845241160643,
	"learning_rate": 2.434468594460022e-05,
	"loss": 0.0903,
	"step": 8760
	},
	{
	"epoch": 0.6830292893539227,
	"grad_norm": 0.2012229390556417,
	"learning_rate": 2.4128755639998368e-05,
	"loss": 0.0994,
	"step": 8780
	},
	{
	"epoch": 0.6845851647283052,
	"grad_norm": 0.19048870485454958,
	"learning_rate": 2.391348225600407e-05,
	"loss": 0.0918,
	"step": 8800
	},
	{
	"epoch": 0.6861410401026877,
	"grad_norm": 0.24637528672036763,
	"learning_rate": 2.3698871258822215e-05,
	"loss": 0.0952,
	"step": 8820
	},
	{
	"epoch": 0.6876969154770702,
	"grad_norm": 0.24744226278491704,
	"learning_rate": 2.34849280978384e-05,
	"loss": 0.1023,
	"step": 8840
	},
	{
	"epoch": 0.6892527908514527,
	"grad_norm": 0.26262665538767777,
	"learning_rate": 2.327165820548059e-05,
	"loss": 0.1109,
	"step": 8860
	},
	{
	"epoch": 0.6908086662258354,
	"grad_norm": 0.16278466551204585,
	"learning_rate": 2.3059066997081162e-05,
	"loss": 0.1039,
	"step": 8880
	},
	{
	"epoch": 0.6923645416002179,
	"grad_norm": 0.2635969659077873,
	"learning_rate": 2.2847159870739416e-05,
	"loss": 0.1144,
	"step": 8900
	},
	{
	"epoch": 0.6939204169746004,
	"grad_norm": 0.21384669127105627,
	"learning_rate": 2.263594220718445e-05,
	"loss": 0.0943,
	"step": 8920
	},
	{
	"epoch": 0.6954762923489829,
	"grad_norm": 0.1760994015919489,
	"learning_rate": 2.2425419369638662e-05,
	"loss": 0.0997,
	"step": 8940
	},
	{
	"epoch": 0.6970321677233654,
	"grad_norm": 0.3312902810570638,
	"learning_rate": 2.2215596703681417e-05,
	"loss": 0.1011,
	"step": 8960
	},
	{
	"epoch": 0.6985880430977479,
	"grad_norm": 0.28202355738694024,
	"learning_rate": 2.2006479537113363e-05,
	"loss": 0.0896,
	"step": 8980
	},
	{
	"epoch": 0.7001439184721304,
	"grad_norm": 0.3374647961310942,
	"learning_rate": 2.17980731798212e-05,
	"loss": 0.1334,
	"step": 9000
	},
	{
	"epoch": 0.7016997938465129,
	"grad_norm": 0.35505279385994787,
	"learning_rate": 2.1590382923642815e-05,
	"loss": 0.096,
	"step": 9020
	},
	{
	"epoch": 0.7032556692208954,
	"grad_norm": 0.28279161572672473,
	"learning_rate": 2.1383414042232898e-05,
	"loss": 0.1076,
	"step": 9040
	},
	{
	"epoch": 0.704811544595278,
	"grad_norm": 0.25911498522087584,
	"learning_rate": 2.117717179092905e-05,
	"loss": 0.0791,
	"step": 9060
	},
	{
	"epoch": 0.7063674199696605,
	"grad_norm": 0.20323675619383785,
	"learning_rate": 2.0971661406618354e-05,
	"loss": 0.0949,
	"step": 9080
	},
	{
	"epoch": 0.707923295344043,
	"grad_norm": 0.3228744363251875,
	"learning_rate": 2.0766888107604348e-05,
	"loss": 0.1134,
	"step": 9100
	},
	{
	"epoch": 0.7094791707184255,
	"grad_norm": 0.22143128418133384,
	"learning_rate": 2.056285709347459e-05,
	"loss": 0.0987,
	"step": 9120
	},
	{
	"epoch": 0.711035046092808,
	"grad_norm": 0.2730176444900543,
	"learning_rate": 2.035957354496859e-05,
	"loss": 0.0953,
	"step": 9140
	},
	{
	"epoch": 0.7125909214671905,
	"grad_norm": 0.42577645292819233,
	"learning_rate": 2.0157042623846246e-05,
	"loss": 0.1043,
	"step": 9160
	},
	{
	"epoch": 0.714146796841573,
	"grad_norm": 0.3872370686158633,
	"learning_rate": 1.9955269472756854e-05,
	"loss": 0.103,
	"step": 9180
	},
	{
	"epoch": 0.7157026722159555,
	"grad_norm": 0.6228197490640249,
	"learning_rate": 1.9754259215108424e-05,
	"loss": 0.0804,
	"step": 9200
	},
	{
	"epoch": 0.717258547590338,
	"grad_norm": 0.14689318864266865,
	"learning_rate": 1.9554016954937653e-05,
	"loss": 0.0995,
	"step": 9220
	},
	{
	"epoch": 0.7188144229647205,
	"grad_norm": 0.17584866832266255,
	"learning_rate": 1.9354547776780303e-05,
	"loss": 0.0947,
	"step": 9240
	},
	{
	"epoch": 0.720370298339103,
	"grad_norm": 0.17884458072993062,
	"learning_rate": 1.9155856745542107e-05,
	"loss": 0.1101,
	"step": 9260
	},
	{
	"epoch": 0.7219261737134856,
	"grad_norm": 0.20875606312457087,
	"learning_rate": 1.8957948906370138e-05,
	"loss": 0.1197,
	"step": 9280
	},
	{
	"epoch": 0.7234820490878681,
	"grad_norm": 0.19284857731857136,
	"learning_rate": 1.876082928452472e-05,
	"loss": 0.0899,
	"step": 9300
	},
	{
	"epoch": 0.7250379244622506,
	"grad_norm": 0.31104430319128357,
	"learning_rate": 1.856450288525185e-05,
	"loss": 0.0994,
	"step": 9320
	},
	{
	"epoch": 0.7265937998366331,
	"grad_norm": 0.2303273671510742,
	"learning_rate": 1.8368974693656038e-05,
	"loss": 0.0982,
	"step": 9340
	},
	{
	"epoch": 0.7281496752110156,
	"grad_norm": 0.1890879549117166,
	"learning_rate": 1.81742496745738e-05,
	"loss": 0.0945,
	"step": 9360
	},
	{
	"epoch": 0.7297055505853981,
	"grad_norm": 0.22765816319943505,
	"learning_rate": 1.798033277244755e-05,
	"loss": 0.0893,
	"step": 9380
	},
	{
	"epoch": 0.7312614259597806,
	"grad_norm": 0.17054843010960036,
	"learning_rate": 1.7787228911200045e-05,
	"loss": 0.119,
	"step": 9400
	},
	{
	"epoch": 0.7328173013341631,
	"grad_norm": 0.1984841392521326,
	"learning_rate": 1.759494299410939e-05,
	"loss": 0.0888,
	"step": 9420
	},
	{
	"epoch": 0.7343731767085456,
	"grad_norm": 0.25412494246879874,
	"learning_rate": 1.7403479903684496e-05,
	"loss": 0.0803,
	"step": 9440
	},
	{
	"epoch": 0.7359290520829281,
	"grad_norm": 0.20501694064898984,
	"learning_rate": 1.721284450154118e-05,
	"loss": 0.1016,
	"step": 9460
	},
	{
	"epoch": 0.7374849274573106,
	"grad_norm": 0.22553830574081873,
	"learning_rate": 1.7023041628278576e-05,
	"loss": 0.0902,
	"step": 9480
	},
	{
	"epoch": 0.7390408028316932,
	"grad_norm": 0.16052559672804065,
	"learning_rate": 1.6834076103356373e-05,
	"loss": 0.077,
	"step": 9500
	},
	{
	"epoch": 0.7405966782060757,
	"grad_norm": 0.16196975040996095,
	"learning_rate": 1.6645952724972354e-05,
	"loss": 0.0993,
	"step": 9520
	},
	{
	"epoch": 0.7421525535804582,
	"grad_norm": 0.26991340152315435,
	"learning_rate": 1.645867626994058e-05,
	"loss": 0.0996,
	"step": 9540
	},
	{
	"epoch": 0.7437084289548407,
	"grad_norm": 0.1778247898029384,
	"learning_rate": 1.6272251493570123e-05,
	"loss": 0.0884,
	"step": 9560
	},
	{
	"epoch": 0.7452643043292232,
	"grad_norm": 0.22112738548467822,
	"learning_rate": 1.6086683129544277e-05,
	"loss": 0.0838,
	"step": 9580
	},
	{
	"epoch": 0.7468201797036057,
	"grad_norm": 0.1668730216162212,
	"learning_rate": 1.5901975889800386e-05,
	"loss": 0.1199,
	"step": 9600
	},
	{
	"epoch": 0.7483760550779882,
	"grad_norm": 0.24196870027762513,
	"learning_rate": 1.571813446441019e-05,
	"loss": 0.0739,
	"step": 9620
	},
	{
	"epoch": 0.7499319304523707,
	"grad_norm": 0.17408724556686292,
	"learning_rate": 1.5535163521460745e-05,
	"loss": 0.0939,
	"step": 9640
	},
	{
	"epoch": 0.7514878058267532,
	"grad_norm": 0.1854157354762736,
	"learning_rate": 1.535306770693588e-05,
	"loss": 0.1038,
	"step": 9660
	},
	{
	"epoch": 0.7530436812011357,
	"grad_norm": 0.13126054665471393,
	"learning_rate": 1.5171851644598228e-05,
	"loss": 0.0855,
	"step": 9680
	},
	{
	"epoch": 0.7545995565755182,
	"grad_norm": 0.21941088587464813,
	"learning_rate": 1.499151993587185e-05,
	"loss": 0.1112,
	"step": 9700
	},
	{
	"epoch": 0.7561554319499008,
	"grad_norm": 0.27246549590974883,
	"learning_rate": 1.4812077159725346e-05,
	"loss": 0.0904,
	"step": 9720
	},
	{
	"epoch": 0.7577113073242834,
	"grad_norm": 0.23883737533925192,
	"learning_rate": 1.4633527872555602e-05,
	"loss": 0.0757,
	"step": 9740
	},
	{
	"epoch": 0.7592671826986659,
	"grad_norm": 0.2555908694273789,
	"learning_rate": 1.4455876608072116e-05,
	"loss": 0.0937,
	"step": 9760
	},
	{
	"epoch": 0.7608230580730484,
	"grad_norm": 0.295451209367461,
	"learning_rate": 1.427912787718187e-05,
	"loss": 0.078,
	"step": 9780
	},
	{
	"epoch": 0.7623789334474309,
	"grad_norm": 0.20680149219645155,
	"learning_rate": 1.4103286167874764e-05,
	"loss": 0.0924,
	"step": 9800
	},
	{
	"epoch": 0.7639348088218134,
	"grad_norm": 0.14753507043246913,
	"learning_rate": 1.3928355945109688e-05,
	"loss": 0.0845,
	"step": 9820
	},
	{
	"epoch": 0.7654906841961959,
	"grad_norm": 0.1799757302999019,
	"learning_rate": 1.3754341650701147e-05,
	"loss": 0.0823,
	"step": 9840
	},
	{
	"epoch": 0.7670465595705784,
	"grad_norm": 0.2558561366715726,
	"learning_rate": 1.3581247703206451e-05,
	"loss": 0.096,
	"step": 9860
	},
	{
	"epoch": 0.7686024349449609,
	"grad_norm": 0.18433952965277892,
	"learning_rate": 1.3409078497813543e-05,
	"loss": 0.0667,
	"step": 9880
	},
	{
	"epoch": 0.7701583103193435,
	"grad_norm": 0.18350612376396253,
	"learning_rate": 1.323783840622937e-05,
	"loss": 0.1156,
	"step": 9900
	},
	{
	"epoch": 0.771714185693726,
	"grad_norm": 0.22465461116463445,
	"learning_rate": 1.306753177656892e-05,
	"loss": 0.0976,
	"step": 9920
	},
	{
	"epoch": 0.7732700610681085,
	"grad_norm": 0.23843987460954788,
	"learning_rate": 1.2898162933244763e-05,
	"loss": 0.102,
	"step": 9940
	},
	{
	"epoch": 0.774825936442491,
	"grad_norm": 0.46338325578159,
	"learning_rate": 1.2729736176857292e-05,
	"loss": 0.1277,
	"step": 9960
	},
	{
	"epoch": 0.7763818118168735,
	"grad_norm": 0.3172884840192611,
	"learning_rate": 1.256225578408549e-05,
	"loss": 0.0872,
	"step": 9980
	},
	{
	"epoch": 0.777937687191256,
	"grad_norm": 0.22784413848662516,
	"learning_rate": 1.2395726007578361e-05,
	"loss": 0.0933,
	"step": 10000
	},
	{
	"epoch": 0.7794935625656385,
	"grad_norm": 0.24087943322072586,
	"learning_rate": 1.2230151075846918e-05,
	"loss": 0.0901,
	"step": 10020
	},
	{
	"epoch": 0.781049437940021,
	"grad_norm": 0.18785941279257512,
	"learning_rate": 1.2065535193156852e-05,
	"loss": 0.0839,
	"step": 10040
	},
	{
	"epoch": 0.7826053133144035,
	"grad_norm": 0.36839791971883906,
	"learning_rate": 1.1901882539421732e-05,
	"loss": 0.1139,
	"step": 10060
	},
	{
	"epoch": 0.784161188688786,
	"grad_norm": 0.16019438171714226,
	"learning_rate": 1.1739197270096914e-05,
	"loss": 0.0836,
	"step": 10080
	},
	{
	"epoch": 0.7857170640631685,
	"grad_norm": 0.2826645872455996,
	"learning_rate": 1.1577483516073989e-05,
	"loss": 0.098,
	"step": 10100
	},
	{
	"epoch": 0.7872729394375511,
	"grad_norm": 0.31165716328505716,
	"learning_rate": 1.1416745383575928e-05,
	"loss": 0.0815,
	"step": 10120
	},
	{
	"epoch": 0.7888288148119336,
	"grad_norm": 0.1995353432738232,
	"learning_rate": 1.1256986954052773e-05,
	"loss": 0.0873,
	"step": 10140
	},
	{
	"epoch": 0.7903846901863161,
	"grad_norm": 0.22704577430722592,
	"learning_rate": 1.1098212284078036e-05,
	"loss": 0.0976,
	"step": 10160
	},
	{
	"epoch": 0.7919405655606986,
	"grad_norm": 0.35969785079208977,
	"learning_rate": 1.0940425405245686e-05,
	"loss": 0.0921,
	"step": 10180
	},
	{
	"epoch": 0.7934964409350811,
	"grad_norm": 0.17151454287291484,
	"learning_rate": 1.0783630324067762e-05,
	"loss": 0.0888,
	"step": 10200
	},
	{
	"epoch": 0.7950523163094636,
	"grad_norm": 0.3237020727953854,
	"learning_rate": 1.0627831021872691e-05,
	"loss": 0.1108,
	"step": 10220
	},
	{
	"epoch": 0.7966081916838461,
	"grad_norm": 0.32735284954903326,
	"learning_rate": 1.047303145470412e-05,
	"loss": 0.0881,
	"step": 10240
	},
	{
	"epoch": 0.7981640670582286,
	"grad_norm": 0.3632770063774064,
	"learning_rate": 1.0319235553220518e-05,
	"loss": 0.0787,
	"step": 10260
	},
	{
	"epoch": 0.7997199424326111,
	"grad_norm": 0.2145794692083885,
	"learning_rate": 1.016644722259536e-05,
	"loss": 0.0866,
	"step": 10280
	},
	{
	"epoch": 0.8012758178069936,
	"grad_norm": 0.21690289738294088,
	"learning_rate": 1.0014670342417936e-05,
	"loss": 0.0895,
	"step": 10300
	},
	{
	"epoch": 0.8028316931813761,
	"grad_norm": 0.2306482803284437,
	"learning_rate": 9.863908766594903e-06,
	"loss": 0.0845,
	"step": 10320
	},
	{
	"epoch": 0.8043875685557587,
	"grad_norm": 0.2586561677178318,
	"learning_rate": 9.71416632325235e-06,
	"loss": 0.1016,
	"step": 10340
	},
	{
	"epoch": 0.8059434439301412,
	"grad_norm": 0.3123656908164633,
	"learning_rate": 9.565446814638668e-06,
	"loss": 0.1048,
	"step": 10360
	},
	{
	"epoch": 0.8074993193045237,
	"grad_norm": 0.2594409253528185,
	"learning_rate": 9.417754017027946e-06,
	"loss": 0.0868,
	"step": 10380
	},
	{
	"epoch": 0.8090551946789062,
	"grad_norm": 0.32137301580238203,
	"learning_rate": 9.271091680624121e-06,
	"loss": 0.0963,
	"step": 10400
	},
	{
	"epoch": 0.8106110700532887,
	"grad_norm": 0.3224913438340862,
	"learning_rate": 9.125463529465733e-06,
	"loss": 0.0953,
	"step": 10420
	},
	{
	"epoch": 0.8121669454276712,
	"grad_norm": 0.19432126550873483,
	"learning_rate": 8.980873261331374e-06,
	"loss": 0.1025,
	"step": 10440
	},
	{
	"epoch": 0.8137228208020537,
	"grad_norm": 0.21949699558768546,
	"learning_rate": 8.837324547645793e-06,
	"loss": 0.0854,
	"step": 10460
	},
	{
	"epoch": 0.8152786961764362,
	"grad_norm": 0.24758859570906425,
	"learning_rate": 8.694821033386663e-06,
	"loss": 0.1073,
	"step": 10480
	},
	{
	"epoch": 0.8168345715508187,
	"grad_norm": 0.1870730002640673,
	"learning_rate": 8.553366336992052e-06,
	"loss": 0.089,
	"step": 10500
	},
	{
	"epoch": 0.8183904469252012,
	"grad_norm": 0.28974822402459943,
	"learning_rate": 8.412964050268507e-06,
	"loss": 0.0866,
	"step": 10520
	},
	{
	"epoch": 0.8199463222995838,
	"grad_norm": 0.34962023314298263,
	"learning_rate": 8.273617738299876e-06,
	"loss": 0.0952,
	"step": 10540
	},
	{
	"epoch": 0.8215021976739663,
	"grad_norm": 0.28192742398031756,
	"learning_rate": 8.135330939356794e-06,
	"loss": 0.0897,
	"step": 10560
	},
	{
	"epoch": 0.8230580730483489,
	"grad_norm": 0.26302243632976813,
	"learning_rate": 7.998107164806806e-06,
	"loss": 0.1,
	"step": 10580
	},
	{
	"epoch": 0.8246139484227314,
	"grad_norm": 0.15501534478513054,
	"learning_rate": 7.861949899025233e-06,
	"loss": 0.0697,
	"step": 10600
	},
	{
	"epoch": 0.8261698237971139,
	"grad_norm": 0.2309776079442696,
	"learning_rate": 7.72686259930669e-06,
	"loss": 0.1035,
	"step": 10620
	},
	{
	"epoch": 0.8277256991714964,
	"grad_norm": 0.1999679968985069,
	"learning_rate": 7.592848695777294e-06,
	"loss": 0.0975,
	"step": 10640
	},
	{
	"epoch": 0.8292815745458789,
	"grad_norm": 0.18788759576672484,
	"learning_rate": 7.459911591307573e-06,
	"loss": 0.0994,
	"step": 10660
	},
	{
	"epoch": 0.8308374499202614,
	"grad_norm": 0.24198652713336835,
	"learning_rate": 7.328054661426053e-06,
	"loss": 0.098,
	"step": 10680
	},
	{
	"epoch": 0.8323933252946439,
	"grad_norm": 0.18875919549267106,
	"learning_rate": 7.197281254233562e-06,
	"loss": 0.0895,
	"step": 10700
	},
	{
	"epoch": 0.8339492006690264,
	"grad_norm": 0.30288189059856974,
	"learning_rate": 7.067594690318196e-06,
	"loss": 0.0851,
	"step": 10720
	},
	{
	"epoch": 0.835505076043409,
	"grad_norm": 0.3101817642069397,
	"learning_rate": 6.938998262670998e-06,
	"loss": 0.1059,
	"step": 10740
	},
	{
	"epoch": 0.8370609514177915,
	"grad_norm": 0.35968181120859927,
	"learning_rate": 6.8114952366024e-06,
	"loss": 0.1015,
	"step": 10760
	},
	{
	"epoch": 0.838616826792174,
	"grad_norm": 0.12745850193723926,
	"learning_rate": 6.6850888496592214e-06,
	"loss": 0.0806,
	"step": 10780
	},
	{
	"epoch": 0.8401727021665565,
	"grad_norm": 0.27775396873658975,
	"learning_rate": 6.559782311542534e-06,
	"loss": 0.0872,
	"step": 10800
	},
	{
	"epoch": 0.841728577540939,
	"grad_norm": 0.17227942458715934,
	"learning_rate": 6.435578804026115e-06,
	"loss": 0.0837,
	"step": 10820
	},
	{
	"epoch": 0.8432844529153215,
	"grad_norm": 0.3851191499835867,
	"learning_rate": 6.312481480875698e-06,
	"loss": 0.1108,
	"step": 10840
	},
	{
	"epoch": 0.844840328289704,
	"grad_norm": 0.3162793004612279,
	"learning_rate": 6.190493467768843e-06,
	"loss": 0.0819,
	"step": 10860
	},
	{
	"epoch": 0.8463962036640865,
	"grad_norm": 0.34409996065769927,
	"learning_rate": 6.069617862215621e-06,
	"loss": 0.0915,
	"step": 10880
	},
	{
	"epoch": 0.847952079038469,
	"grad_norm": 0.330570114721541,
	"learning_rate": 5.949857733479919e-06,
	"loss": 0.1007,
	"step": 10900
	},
	{
	"epoch": 0.8495079544128515,
	"grad_norm": 0.3467228433108101,
	"learning_rate": 5.8312161225015375e-06,
	"loss": 0.1003,
	"step": 10920
	},
	{
	"epoch": 0.851063829787234,
	"grad_norm": 0.20179445586948613,
	"learning_rate": 5.713696041818961e-06,
	"loss": 0.0889,
	"step": 10940
	},
	{
	"epoch": 0.8526197051616166,
	"grad_norm": 0.2959782219218903,
	"learning_rate": 5.59730047549285e-06,
	"loss": 0.1061,
	"step": 10960
	},
	{
	"epoch": 0.8541755805359991,
	"grad_norm": 0.3202205272910113,
	"learning_rate": 5.482032379030299e-06,
	"loss": 0.0909,
	"step": 10980
	},
	{
	"epoch": 0.8557314559103816,
	"grad_norm": 0.10897108726271473,
	"learning_rate": 5.367894679309782e-06,
	"loss": 0.1107,
	"step": 11000
	},
	{
	"epoch": 0.8572873312847641,
	"grad_norm": 0.15360504201990405,
	"learning_rate": 5.254890274506802e-06,
	"loss": 0.091,
	"step": 11020
	},
	{
	"epoch": 0.8588432066591466,
	"grad_norm": 0.20019589264276277,
	"learning_rate": 5.143022034020351e-06,
	"loss": 0.1008,
	"step": 11040
	},
	{
	"epoch": 0.8603990820335291,
	"grad_norm": 0.20965315133351023,
	"learning_rate": 5.032292798400018e-06,
	"loss": 0.0991,
	"step": 11060
	},
	{
	"epoch": 0.8619549574079116,
	"grad_norm": 0.19327840948864372,
	"learning_rate": 4.922705379273862e-06,
	"loss": 0.0817,
	"step": 11080
	},
	{
	"epoch": 0.8635108327822941,
	"grad_norm": 0.3842523683291388,
	"learning_rate": 4.814262559277038e-06,
	"loss": 0.0794,
	"step": 11100
	},
	{
	"epoch": 0.8650667081566766,
	"grad_norm": 0.22892079191559764,
	"learning_rate": 4.7069670919811294e-06,
	"loss": 0.0868,
	"step": 11120
	},
	{
	"epoch": 0.8666225835310591,
	"grad_norm": 0.37205444542545973,
	"learning_rate": 4.6008217018242206e-06,
	"loss": 0.1004,
	"step": 11140
	},
	{
	"epoch": 0.8681784589054417,
	"grad_norm": 0.2158369767423703,
	"learning_rate": 4.495829084041736e-06,
	"loss": 0.1069,
	"step": 11160
	},
	{
	"epoch": 0.8697343342798242,
	"grad_norm": 0.24779277579385603,
	"learning_rate": 4.3919919045979865e-06,
	"loss": 0.0979,
	"step": 11180
	},
	{
	"epoch": 0.8712902096542067,
	"grad_norm": 0.3466201291586451,
	"learning_rate": 4.289312800118494e-06,
	"loss": 0.0812,
	"step": 11200
	},
	{
	"epoch": 0.8728460850285892,
	"grad_norm": 0.17342568425849422,
	"learning_rate": 4.187794377823018e-06,
	"loss": 0.0778,
	"step": 11220
	},
	{
	"epoch": 0.8744019604029717,
	"grad_norm": 0.1793971497332504,
	"learning_rate": 4.087439215459388e-06,
	"loss": 0.0995,
	"step": 11240
	},
	{
	"epoch": 0.8759578357773542,
	"grad_norm": 0.16346310698824007,
	"learning_rate": 3.988249861237991e-06,
	"loss": 0.0913,
	"step": 11260
	},
	{
	"epoch": 0.8775137111517367,
	"grad_norm": 0.18392625746781902,
	"learning_rate": 3.890228833767156e-06,
	"loss": 0.0968,
	"step": 11280
	},
	{
	"epoch": 0.8790695865261192,
	"grad_norm": 0.244692815533656,
	"learning_rate": 3.7933786219891132e-06,
	"loss": 0.0958,
	"step": 11300
	},
	{
	"epoch": 0.8806254619005017,
	"grad_norm": 0.2688844546854025,
	"learning_rate": 3.6977016851168446e-06,
	"loss": 0.0957,
	"step": 11320
	},
	{
	"epoch": 0.8821813372748842,
	"grad_norm": 0.18980545171193855,
	"learning_rate": 3.6032004525716257e-06,
	"loss": 0.0854,
	"step": 11340
	},
	{
	"epoch": 0.8837372126492667,
	"grad_norm": 0.12796592391644104,
	"learning_rate": 3.5098773239213402e-06,
	"loss": 0.0789,
	"step": 11360
	},
	{
	"epoch": 0.8852930880236493,
	"grad_norm": 0.17586930835200434,
	"learning_rate": 3.41773466881955e-06,
	"loss": 0.0967,
	"step": 11380
	},
	{
	"epoch": 0.8868489633980318,
	"grad_norm": 0.17002464986011556,
	"learning_rate": 3.3267748269453337e-06,
	"loss": 0.102,
	"step": 11400
	},
	{
	"epoch": 0.8884048387724144,
	"grad_norm": 0.17731707769572266,
	"learning_rate": 3.2370001079438494e-06,
	"loss": 0.0832,
	"step": 11420
	},
	{
	"epoch": 0.8899607141467969,
	"grad_norm": 0.360527536204551,
	"learning_rate": 3.148412791367733e-06,
	"loss": 0.1074,
	"step": 11440
	},
	{
	"epoch": 0.8915165895211794,
	"grad_norm": 0.16595849639263094,
	"learning_rate": 3.061015126619171e-06,
	"loss": 0.095,
	"step": 11460
	},
	{
	"epoch": 0.8930724648955619,
	"grad_norm": 0.14078037449616376,
	"learning_rate": 2.9748093328928086e-06,
	"loss": 0.0941,
	"step": 11480
	},
	{
	"epoch": 0.8946283402699444,
	"grad_norm": 0.24152506888614694,
	"learning_rate": 2.8897975991194093e-06,
	"loss": 0.0829,
	"step": 11500
	},
	{
	"epoch": 0.8961842156443269,
	"grad_norm": 0.16687093757615304,
	"learning_rate": 2.8059820839102413e-06,
	"loss": 0.0774,
	"step": 11520
	},
	{
	"epoch": 0.8977400910187094,
	"grad_norm": 0.18218681761372854,
	"learning_rate": 2.7233649155022913e-06,
	"loss": 0.0916,
	"step": 11540
	},
	{
	"epoch": 0.899295966393092,
	"grad_norm": 0.2601712004435821,
	"learning_rate": 2.6419481917042245e-06,
	"loss": 0.0896,
	"step": 11560
	},
	{
	"epoch": 0.9008518417674745,
	"grad_norm": 0.2647587129970373,
	"learning_rate": 2.5617339798431007e-06,
	"loss": 0.1012,
	"step": 11580
	},
	{
	"epoch": 0.902407717141857,
	"grad_norm": 0.31662337518822964,
	"learning_rate": 2.482724316711904e-06,
	"loss": 0.0936,
	"step": 11600
	},
	{
	"epoch": 0.9039635925162395,
	"grad_norm": 0.2897408119458106,
	"learning_rate": 2.4049212085177907e-06,
	"loss": 0.0845,
	"step": 11620
	},
	{
	"epoch": 0.905519467890622,
	"grad_norm": 0.2503978112126087,
	"learning_rate": 2.3283266308311956e-06,
	"loss": 0.0827,
	"step": 11640
	},
	{
	"epoch": 0.9070753432650045,
	"grad_norm": 0.4241488376493025,
	"learning_rate": 2.2529425285356176e-06,
	"loss": 0.0908,
	"step": 11660
	},
	{
	"epoch": 0.908631218639387,
	"grad_norm": 0.18270244667898686,
	"learning_rate": 2.1787708157782694e-06,
	"loss": 0.0865,
	"step": 11680
	},
	{
	"epoch": 0.9101870940137695,
	"grad_norm": 0.21606905884865651,
	"learning_rate": 2.1058133759214725e-06,
	"loss": 0.103,
	"step": 11700
	},
	{
	"epoch": 0.911742969388152,
	"grad_norm": 0.24585590613749583,
	"learning_rate": 2.0340720614948073e-06,
	"loss": 0.1016,
	"step": 11720
	},
	{
	"epoch": 0.9132988447625345,
	"grad_norm": 0.2008613104589357,
	"learning_rate": 1.96354869414811e-06,
	"loss": 0.0898,
	"step": 11740
	},
	{
	"epoch": 0.914854720136917,
	"grad_norm": 0.18124232395227519,
	"learning_rate": 1.8942450646051824e-06,
	"loss": 0.1087,
	"step": 11760
	},
	{
	"epoch": 0.9164105955112996,
	"grad_norm": 0.3487752382638369,
	"learning_rate": 1.8261629326183639e-06,
	"loss": 0.0699,
	"step": 11780
	},
	{
	"epoch": 0.9179664708856821,
	"grad_norm": 0.2324256412706674,
	"learning_rate": 1.759304026923797e-06,
	"loss": 0.0833,
	"step": 11800
	},
	{
	"epoch": 0.9195223462600646,
	"grad_norm": 0.16857848872345094,
	"learning_rate": 1.6936700451975817e-06,
	"loss": 0.0971,
	"step": 11820
	},
	{
	"epoch": 0.9210782216344471,
	"grad_norm": 0.32589655203087087,
	"learning_rate": 1.6292626540126255e-06,
	"loss": 0.1203,
	"step": 11840
	},
	{
	"epoch": 0.9226340970088296,
	"grad_norm": 0.27370225897645545,
	"learning_rate": 1.5660834887963604e-06,
	"loss": 0.0881,
	"step": 11860
	},
	{
	"epoch": 0.9241899723832121,
	"grad_norm": 0.24743829754861812,
	"learning_rate": 1.5041341537891885e-06,
	"loss": 0.0805,
	"step": 11880
	},
	{
	"epoch": 0.9257458477575946,
	"grad_norm": 0.27786891830919325,
	"learning_rate": 1.443416222003774e-06,
	"loss": 0.0819,
	"step": 11900
	},
	{
	"epoch": 0.9273017231319771,
	"grad_norm": 0.3242798952029978,
	"learning_rate": 1.3839312351850654e-06,
	"loss": 0.0904,
	"step": 11920
	},
	{
	"epoch": 0.9288575985063596,
	"grad_norm": 0.2768221769111869,
	"learning_rate": 1.3256807037711871e-06,
	"loss": 0.1028,
	"step": 11940
	},
	{
	"epoch": 0.9304134738807421,
	"grad_norm": 0.17359797685515962,
	"learning_rate": 1.268666106855071e-06,
	"loss": 0.0972,
	"step": 11960
	},
	{
	"epoch": 0.9319693492551246,
	"grad_norm": 0.21206613086374923,
	"learning_rate": 1.212888892146885e-06,
	"loss": 0.0786,
	"step": 11980
	},
	{
	"epoch": 0.9335252246295072,
	"grad_norm": 0.31422667328212367,
	"learning_rate": 1.1583504759372922e-06,
	"loss": 0.0924,
	"step": 12000
	},
	{
	"epoch": 0.9350811000038897,
	"grad_norm": 0.2528657385645779,
	"learning_rate": 1.1050522430614663e-06,
	"loss": 0.0902,
	"step": 12020
	},
	{
	"epoch": 0.9366369753782722,
	"grad_norm": 0.24010727671682566,
	"learning_rate": 1.0529955468639663e-06,
	"loss": 0.0937,
	"step": 12040
	},
	{
	"epoch": 0.9381928507526547,
	"grad_norm": 0.2227702611419707,
	"learning_rate": 1.002181709164346e-06,
	"loss": 0.0891,
	"step": 12060
	},
	{
	"epoch": 0.9397487261270372,
	"grad_norm": 0.24627854631066437,
	"learning_rate": 9.526120202235756e-07,
	"loss": 0.0922,
	"step": 12080
	},
	{
	"epoch": 0.9413046015014197,
	"grad_norm": 0.23573398406446816,
	"learning_rate": 9.042877387113125e-07,
	"loss": 0.0923,
	"step": 12100
	},
	{
	"epoch": 0.9428604768758022,
	"grad_norm": 0.5010107950391143,
	"learning_rate": 8.572100916739267e-07,
	"loss": 0.0876,
	"step": 12120
	},
	{
	"epoch": 0.9444163522501847,
	"grad_norm": 0.23830017610386595,
	"learning_rate": 8.113802745033316e-07,
	"loss": 0.1004,
	"step": 12140
	},
	{
	"epoch": 0.9459722276245672,
	"grad_norm": 0.20667313180215402,
	"learning_rate": 7.667994509066522e-07,
	"loss": 0.0773,
	"step": 12160
	},
	{
	"epoch": 0.9475281029989497,
	"grad_norm": 0.205698963767961,
	"learning_rate": 7.234687528766715e-07,
	"loss": 0.0911,
	"step": 12180
	},
	{
	"epoch": 0.9490839783733322,
	"grad_norm": 0.23606053401444732,
	"learning_rate": 6.81389280663064e-07,
	"loss": 0.1031,
	"step": 12200
	},
	{
	"epoch": 0.9506398537477148,
	"grad_norm": 0.2461291507323295,
	"learning_rate": 6.405621027444963e-07,
	"loss": 0.0846,
	"step": 12220
	},
	{
	"epoch": 0.9521957291220973,
	"grad_norm": 0.2678649514539114,
	"learning_rate": 6.009882558014601e-07,
	"loss": 0.0944,
	"step": 12240
	},
	{
	"epoch": 0.9537516044964798,
	"grad_norm": 0.4180134882006999,
	"learning_rate": 5.626687446899814e-07,
	"loss": 0.095,
	"step": 12260
	},
	{
	"epoch": 0.9553074798708624,
	"grad_norm": 0.1557176730573096,
	"learning_rate": 5.256045424160749e-07,
	"loss": 0.1013,
	"step": 12280
	},
	{
	"epoch": 0.9568633552452449,
	"grad_norm": 0.13827103011190442,
	"learning_rate": 4.897965901110635e-07,
	"loss": 0.0927,
	"step": 12300
	},
	{
	"epoch": 0.9584192306196274,
	"grad_norm": 0.15528840747035424,
	"learning_rate": 4.552457970076529e-07,
	"loss": 0.0807,
	"step": 12320
	},
	{
	"epoch": 0.9599751059940099,
	"grad_norm": 0.3412135622958878,
	"learning_rate": 4.219530404168837e-07,
	"loss": 0.115,
	"step": 12340
	},
	{
	"epoch": 0.9615309813683924,
	"grad_norm": 0.1649898534177822,
	"learning_rate": 3.8991916570580987e-07,
	"loss": 0.0757,
	"step": 12360
	},
	{
	"epoch": 0.9630868567427749,
	"grad_norm": 0.17461620138968847,
	"learning_rate": 3.591449862760721e-07,
	"loss": 0.1113,
	"step": 12380
	},
	{
	"epoch": 0.9646427321171575,
	"grad_norm": 0.2634060430218938,
	"learning_rate": 3.2963128354321913e-07,
	"loss": 0.0898,
	"step": 12400
	},
	{
	"epoch": 0.96619860749154,
	"grad_norm": 0.29130697424865754,
	"learning_rate": 3.0137880691687995e-07,
	"loss": 0.0836,
	"step": 12420
	},
	{
	"epoch": 0.9677544828659225,
	"grad_norm": 0.3128965201028219,
	"learning_rate": 2.743882737817116e-07,
	"loss": 0.0784,
	"step": 12440
	},
	{
	"epoch": 0.969310358240305,
	"grad_norm": 0.19297736091497,
	"learning_rate": 2.486603694792311e-07,
	"loss": 0.0971,
	"step": 12460
	},
	{
	"epoch": 0.9708662336146875,
	"grad_norm": 0.17677998897137398,
	"learning_rate": 2.2419574729035108e-07,
	"loss": 0.0786,
	"step": 12480
	},
	{
	"epoch": 0.97242210898907,
	"grad_norm": 0.13858204711782496,
	"learning_rate": 2.0099502841884888e-07,
	"loss": 0.0991,
	"step": 12500
	},
	{
	"epoch": 0.9739779843634525,
	"grad_norm": 0.28876942028667324,
	"learning_rate": 1.7905880197556234e-07,
	"loss": 0.1082,
	"step": 12520
	},
	{
	"epoch": 0.975533859737835,
	"grad_norm": 0.39608026790984724,
	"learning_rate": 1.5838762496341862e-07,
	"loss": 0.1164,
	"step": 12540
	},
	{
	"epoch": 0.9770897351122175,
	"grad_norm": 0.2800584450516047,
	"learning_rate": 1.3898202226333423e-07,
	"loss": 0.0896,
	"step": 12560
	},
	{
	"epoch": 0.9786456104866,
	"grad_norm": 0.2083443716926576,
	"learning_rate": 1.2084248662084797e-07,
	"loss": 0.0899,
	"step": 12580
	},
	{
	"epoch": 0.9802014858609825,
	"grad_norm": 0.18297757016745886,
	"learning_rate": 1.0396947863362539e-07,
	"loss": 0.085,
	"step": 12600
	},
	{
	"epoch": 0.9817573612353651,
	"grad_norm": 0.13261855186376548,
	"learning_rate": 8.836342673976261e-08,
	"loss": 0.0897,
	"step": 12620
	},
	{
	"epoch": 0.9833132366097476,
	"grad_norm": 0.3104622948337489,
	"learning_rate": 7.402472720690057e-08,
	"loss": 0.1016,
	"step": 12640
	},
	{
	"epoch": 0.9848691119841301,
	"grad_norm": 0.18279371997772798,
	"learning_rate": 6.095374412218302e-08,
	"loss": 0.1066,
	"step": 12660
	},
	{
	"epoch": 0.9864249873585126,
	"grad_norm": 0.19526385403456245,
	"learning_rate": 4.915080938298067e-08,
	"loss": 0.0817,
	"step": 12680
	},
	{
	"epoch": 0.9879808627328951,
	"grad_norm": 0.18246796037367954,
	"learning_rate": 3.861622268848675e-08,
	"loss": 0.0797,
	"step": 12700
	},
	{
	"epoch": 0.9895367381072776,
	"grad_norm": 0.20688223446302664,
	"learning_rate": 2.9350251532100915e-08,
	"loss": 0.0915,
	"step": 12720
	},
	{
	"epoch": 0.9910926134816601,
	"grad_norm": 0.166271985077419,
	"learning_rate": 2.1353131194634666e-08,
	"loss": 0.0746,
	"step": 12740
	},
	{
	"epoch": 0.9926484888560426,
	"grad_norm": 0.39651370643195233,
	"learning_rate": 1.4625064738343907e-08,
	"loss": 0.0834,
	"step": 12760
	},
	{
	"epoch": 0.9942043642304251,
	"grad_norm": 0.1900639074435354,
	"learning_rate": 9.166223001760843e-09,
	"loss": 0.0852,
	"step": 12780
	},
	{
	"epoch": 0.9957602396048076,
	"grad_norm": 0.21688465641141705,
	"learning_rate": 4.9767445953696755e-09,
	"loss": 0.1012,
	"step": 12800
	},
	{
	"epoch": 0.9973161149791901,
	"grad_norm": 0.26574437455875827,
	"learning_rate": 2.0567358980705388e-09,
	"loss": 0.0853,
	"step": 12820
	},
	{
	"epoch": 0.9988719903535727,
	"grad_norm": 0.19229734554227748,
	"learning_rate": 4.062710545038595e-10,
	"loss": 0.0836,
	"step": 12840
	},
	{
	"epoch": 1.0,
	"step": 12855,
	"total_flos": 346853040193536.0,
	"train_loss": 0.12452768658163574,
	"train_runtime": 31982.64,
	"train_samples_per_second": 3.215,
	"train_steps_per_second": 0.402
	}
	],
	"logging_steps": 20,
	"max_steps": 12855,
	"num_input_tokens_seen": 0,
	"num_train_epochs": 1,
	"save_steps": 1000,
	"stateful_callbacks": {
	"TrainerControl": {
	"args": {
	"should_epoch_stop": false,
	"should_evaluate": false,
	"should_log": false,
	"should_save": true,
	"should_training_stop": true
	},
	"attributes": {}
	}
	},
	"total_flos": 346853040193536.0,
	"train_batch_size": 1,
	"trial_name": null,
	"trial_params": null
	}