Super-squash branch 'main' using huggingface_hub

.gitattributes

@@ -0,0 +1,35 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,89 @@
+---
+library_name: transformers
+language:
+- en
+license: apache-2.0
+base_model: facebook/bart-large
+tags:
+- map-reduce
+- summarization
+datasets:
+- pszemraj/summary-map-reduce-v1
+pipeline_tag: text2text-generation
+thumbnail: >-
+  https://cdn-uploads.huggingface.co/production/uploads/60bccec062080d33f875cd0c/Sv7_-MM901qNkyHuBdTC_.png
+---
+
+# bart-large-summary-map-reduce
+
+A text2text model to "map-reduce" summaries of a chunked long document into one.
+
+An [explanation](https://github.com/pszemraj/textsum/wiki/consolidating-summaries) of this model's role as a post-processor for [textsum](https://github.com/pszemraj/textsum) (_or any other long-doc summarization method similar to the below_):
+
+![image/png](https://cdn-uploads.huggingface.co/production/uploads/60bccec062080d33f875cd0c/Sv7_-MM901qNkyHuBdTC_.png)
+
+<small> modified flowchart from Google's blog [here](https://cloud.google.com/blog/products/ai-machine-learning/long-document-summarization-with-workflows-and-gemini-models) </small>
+
+## Details
+This model is a fine-tuned version of [facebook/bart-large](https://huggingface.co/facebook/bart-large) on the pszemraj/summary-map-reduce dataset.
+It achieves the following results on the evaluation set:
+- Loss: 0.7894
+- Num Input Tokens Seen: 14258488
+
+
+## usage 
+
+> [!TIP]
+> BART supports several speedups for inference on GPU, including [flash-attention2](https://huggingface.co/docs/transformers/perf_infer_gpu_one#flashattention-2) and [torch SDPA](https://huggingface.co/docs/transformers/perf_infer_gpu_one#pytorch-scaled-dot-product-attention)
+
+an example of aggregating summaries from chunks of a long document:
+
+```py
+import torch
+from transformers import pipeline
+
+pipe = pipeline(
+    "text2text-generation",
+    model="pszemraj/bart-large-summary-map-reduce",
+    device_map="auto",
+)
+
+# examples
+text = """"Sangers on a Train" is a 1950 film about a train driver, Guy Haines, who discovers his wife, Miriam, has been murdered in Metcalf, Washington, DC. The film delves into the relationship between Guy and Anne Burton, focusing on Guy's desire for Anne to marry him.
+"Screentalk" is a comedy about Anne Burton and her husband, Guy Haines, who are investigating the murder of their daughter, Miriam. The plot revolves around Anne's relationship with Bruno, who has been arrested for his wife's murder. In the second set, Guy and Anne meet at a tennis court in Washington, DC, where they plan to play against each other. Hennessy and Hammond investigate the crime scene, leading to Guy's arrest.
+"The Announcer's Boom Forest Hills" is a tennis game between Guy Haines and Bruno Antony, with the score six-five. In the second set, Haines leads three games to four, but his opponent, Bernard Reynolds, attacks him in the third set. Meanwhile, Anne Hennessy and Barbara Hammond are preparing for dinner at the amusement park, where Guy has been waiting for hours. A police car arrives, followed by a taxi. The boatman and detectives follow Guy through the queue, leading to the conclusion that Guy was the man responsible for the accident."""
+
+text = """A computer implemented method of generating a syntactic object. The method includes the steps of providing a plurality of input data sets, each input data set comprising one or more words, wherein each word is associated with at least one non-adjacent second word; creating an exocentric relationship between the first and second words by applying a neo-ian event semantics to the input data in such a way that the neo-antagonistic effect results in the generation of the syntactic object; and storing the generated syntactic object for future use.
+    A method of learning and using language is disclosed. The method includes the steps of creating a lexicon of words, wherein each word in the lexicon has at least two possible states, selecting a set of one or more of the possible states of the lexicon to be used as a base state for a subsequent computational operation, and applying the computational operation to the base state to form a new output state.
+    A computer implemented method for changing a first workspace to a second workspace. The method includes the steps of creating a new workspace by merging the first workspace with the second workspace, wherein the merging is based on at least one of: an impenetrable condition; a constraint on movement; and a resource restriction.
+    The brain is constantly loosing neurons because you doesn&#39;t want all the junk around."""
+
+# generate
+if torch.cuda.is_available():
+    torch.cuda.empty_cache()
+res = pipe(
+    text,
+    max_new_tokens=512, # increase up to 1024 if needed
+    num_beams=4,
+    early_stopping=True,
+    truncation=True,
+)
+print(res[0]["generated_text"])
+```
+
+
+## Training procedure
+
+### Training hyperparameters
+
+The following hyperparameters were used during training:
+- learning_rate: 0.0001
+- train_batch_size: 4
+- eval_batch_size: 4
+- seed: 17868
+- gradient_accumulation_steps: 16
+- total_train_batch_size: 64
+- optimizer: Use OptimizerNames.PAGED_ADAMW with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
+- lr_scheduler_type: cosine
+- lr_scheduler_warmup_ratio: 0.05
+- num_epochs: 3.0

all_results.json

@@ -0,0 +1,15 @@
+{
+    "epoch": 2.9906542056074765,
+    "eval_loss": 0.78944993019104,
+    "eval_runtime": 0.9197,
+    "eval_samples": 150,
+    "eval_samples_per_second": 163.104,
+    "eval_steps_per_second": 41.32,
+    "num_input_tokens_seen": 14258488,
+    "total_flos": 3.0175424769490944e+16,
+    "train_loss": 0.895275863011678,
+    "train_runtime": 860.318,
+    "train_samples": 16692,
+    "train_samples_per_second": 58.206,
+    "train_steps_per_second": 0.907
+}

config.json

@@ -0,0 +1,73 @@
+{
+  "_name_or_path": "facebook/bart-large",
+  "activation_dropout": 0.1,
+  "activation_function": "gelu",
+  "add_bias_logits": false,
+  "add_final_layer_norm": false,
+  "architectures": [
+    "BartForConditionalGeneration"
+  ],
+  "attention_dropout": 0.1,
+  "bos_token_id": 0,
+  "classif_dropout": 0.1,
+  "classifier_dropout": 0.0,
+  "d_model": 1024,
+  "decoder_attention_heads": 16,
+  "decoder_ffn_dim": 4096,
+  "decoder_layerdrop": 0.0,
+  "decoder_layers": 12,
+  "decoder_start_token_id": 2,
+  "dropout": 0.1,
+  "early_stopping": null,
+  "encoder_attention_heads": 16,
+  "encoder_ffn_dim": 4096,
+  "encoder_layerdrop": 0.0,
+  "encoder_layers": 12,
+  "eos_token_id": 2,
+  "forced_eos_token_id": 2,
+  "gradient_checkpointing": false,
+  "id2label": {
+    "0": "LABEL_0",
+    "1": "LABEL_1",
+    "2": "LABEL_2"
+  },
+  "init_std": 0.02,
+  "is_encoder_decoder": true,
+  "label2id": {
+    "LABEL_0": 0,
+    "LABEL_1": 1,
+    "LABEL_2": 2
+  },
+  "max_position_embeddings": 1024,
+  "model_type": "bart",
+  "no_repeat_ngram_size": null,
+  "normalize_before": false,
+  "num_beams": null,
+  "num_hidden_layers": 12,
+  "pad_token_id": 1,
+  "scale_embedding": false,
+  "task_specific_params": {
+    "summarization": {
+      "length_penalty": 1.0,
+      "max_length": 128,
+      "min_length": 12,
+      "num_beams": 4
+    },
+    "summarization_cnn": {
+      "length_penalty": 2.0,
+      "max_length": 142,
+      "min_length": 56,
+      "num_beams": 4
+    },
+    "summarization_xsum": {
+      "length_penalty": 1.0,
+      "max_length": 62,
+      "min_length": 11,
+      "num_beams": 6
+    }
+  },
+  "torch_dtype": "float32",
+  "transformers_version": "4.46.0.dev0",
+  "use_cache": true,
+  "vocab_size": 50304
+}

eval_results.json

@@ -0,0 +1,9 @@
+{
+    "epoch": 2.9906542056074765,
+    "eval_loss": 0.78944993019104,
+    "eval_runtime": 0.9197,
+    "eval_samples": 150,
+    "eval_samples_per_second": 163.104,
+    "eval_steps_per_second": 41.32,
+    "num_input_tokens_seen": 14258488
+}

generation_config.json

@@ -0,0 +1,13 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 0,
+  "decoder_start_token_id": 2,
+  "early_stopping": true,
+  "eos_token_id": 2,
+  "forced_bos_token_id": 0,
+  "forced_eos_token_id": 2,
+  "no_repeat_ngram_size": 3,
+  "num_beams": 4,
+  "pad_token_id": 1,
+  "transformers_version": "4.46.0.dev0"
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:59614eeaadafdb8d5df62cf77270bca4d4dbd9a8c82e88e0957148ac7f682cfc
+size 1625586896

special_tokens_map.json

@@ -0,0 +1,15 @@
+{
+  "bos_token": "<s>",
+  "cls_token": "<s>",
+  "eos_token": "</s>",
+  "mask_token": {
+    "content": "<mask>",
+    "lstrip": true,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": "<pad>",
+  "sep_token": "</s>",
+  "unk_token": "<unk>"
+}

tokenizer_config.json

@@ -0,0 +1,57 @@
+{
+  "add_prefix_space": false,
+  "added_tokens_decoder": {
+    "0": {
+      "content": "<s>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "1": {
+      "content": "<pad>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "2": {
+      "content": "</s>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "3": {
+      "content": "<unk>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "50264": {
+      "content": "<mask>",
+      "lstrip": true,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "bos_token": "<s>",
+  "clean_up_tokenization_spaces": false,
+  "cls_token": "<s>",
+  "eos_token": "</s>",
+  "errors": "replace",
+  "mask_token": "<mask>",
+  "model_max_length": 1024,
+  "pad_token": "<pad>",
+  "sep_token": "</s>",
+  "tokenizer_class": "BartTokenizer",
+  "trim_offsets": true,
+  "unk_token": "<unk>"
+}

train_results.json

@@ -0,0 +1,10 @@
+{
+    "epoch": 2.9906542056074765,
+    "num_input_tokens_seen": 14258488,
+    "total_flos": 3.0175424769490944e+16,
+    "train_loss": 0.895275863011678,
+    "train_runtime": 860.318,
+    "train_samples": 16692,
+    "train_samples_per_second": 58.206,
+    "train_steps_per_second": 0.907
+}

trainer_state.json

@@ -0,0 +1,1354 @@
+{
+  "best_metric": null,
+  "best_model_checkpoint": null,
+  "epoch": 2.9906542056074765,
+  "eval_steps": 100,
+  "global_step": 780,
+  "is_hyper_param_search": false,
+  "is_local_process_zero": true,
+  "is_world_process_zero": true,
+  "log_history": [
+    {
+      "epoch": 0.01917086029235562,
+      "grad_norm": 10.80891227722168,
+      "learning_rate": 1.282051282051282e-05,
+      "loss": 2.5867,
+      "num_input_tokens_seen": 85900,
+      "step": 5
+    },
+    {
+      "epoch": 0.03834172058471124,
+      "grad_norm": 5.320851802825928,
+      "learning_rate": 2.564102564102564e-05,
+      "loss": 1.778,
+      "num_input_tokens_seen": 168784,
+      "step": 10
+    },
+    {
+      "epoch": 0.05751258087706686,
+      "grad_norm": 2.427093982696533,
+      "learning_rate": 3.846153846153846e-05,
+      "loss": 1.5006,
+      "num_input_tokens_seen": 262708,
+      "step": 15
+    },
+    {
+      "epoch": 0.07668344116942248,
+      "grad_norm": 2.143993854522705,
+      "learning_rate": 5.128205128205128e-05,
+      "loss": 1.3452,
+      "num_input_tokens_seen": 350572,
+      "step": 20
+    },
+    {
+      "epoch": 0.09585430146177809,
+      "grad_norm": 2.46453595161438,
+      "learning_rate": 6.410256410256412e-05,
+      "loss": 1.2953,
+      "num_input_tokens_seen": 441976,
+      "step": 25
+    },
+    {
+      "epoch": 0.11502516175413371,
+      "grad_norm": 2.382268190383911,
+      "learning_rate": 7.692307692307693e-05,
+      "loss": 1.3104,
+      "num_input_tokens_seen": 525052,
+      "step": 30
+    },
+    {
+      "epoch": 0.13419602204648934,
+      "grad_norm": 2.990694761276245,
+      "learning_rate": 8.974358974358975e-05,
+      "loss": 1.306,
+      "num_input_tokens_seen": 626524,
+      "step": 35
+    },
+    {
+      "epoch": 0.15336688233884496,
+      "grad_norm": 1.8448060750961304,
+      "learning_rate": 9.99995506314361e-05,
+      "loss": 1.2473,
+      "num_input_tokens_seen": 709868,
+      "step": 40
+    },
+    {
+      "epoch": 0.17253774263120059,
+      "grad_norm": 2.6468124389648438,
+      "learning_rate": 9.998382357979809e-05,
+      "loss": 1.2091,
+      "num_input_tokens_seen": 823312,
+      "step": 45
+    },
+    {
+      "epoch": 0.19170860292355618,
+      "grad_norm": 1.919690728187561,
+      "learning_rate": 9.994563617659665e-05,
+      "loss": 1.1985,
+      "num_input_tokens_seen": 915100,
+      "step": 50
+    },
+    {
+      "epoch": 0.2108794632159118,
+      "grad_norm": 2.13997483253479,
+      "learning_rate": 9.988500558143337e-05,
+      "loss": 1.1922,
+      "num_input_tokens_seen": 1014472,
+      "step": 55
+    },
+    {
+      "epoch": 0.23005032350826743,
+      "grad_norm": 1.7454973459243774,
+      "learning_rate": 9.980195903881232e-05,
+      "loss": 1.2273,
+      "num_input_tokens_seen": 1111288,
+      "step": 60
+    },
+    {
+      "epoch": 0.24922118380062305,
+      "grad_norm": 2.0489871501922607,
+      "learning_rate": 9.969653386589748e-05,
+      "loss": 1.1729,
+      "num_input_tokens_seen": 1193792,
+      "step": 65
+    },
+    {
+      "epoch": 0.2683920440929787,
+      "grad_norm": 1.851652979850769,
+      "learning_rate": 9.956877743574438e-05,
+      "loss": 1.188,
+      "num_input_tokens_seen": 1293416,
+      "step": 70
+    },
+    {
+      "epoch": 0.2875629043853343,
+      "grad_norm": 2.3986475467681885,
+      "learning_rate": 9.94187471560127e-05,
+      "loss": 1.1459,
+      "num_input_tokens_seen": 1383712,
+      "step": 75
+    },
+    {
+      "epoch": 0.3067337646776899,
+      "grad_norm": 1.6811550855636597,
+      "learning_rate": 9.924651044317017e-05,
+      "loss": 1.1561,
+      "num_input_tokens_seen": 1483596,
+      "step": 80
+    },
+    {
+      "epoch": 0.3259046249700455,
+      "grad_norm": 2.3249671459198,
+      "learning_rate": 9.90521446921987e-05,
+      "loss": 1.1224,
+      "num_input_tokens_seen": 1561392,
+      "step": 85
+    },
+    {
+      "epoch": 0.34507548526240117,
+      "grad_norm": 2.17244553565979,
+      "learning_rate": 9.883573724181683e-05,
+      "loss": 1.1283,
+      "num_input_tokens_seen": 1660352,
+      "step": 90
+    },
+    {
+      "epoch": 0.36424634555475677,
+      "grad_norm": 2.0246732234954834,
+      "learning_rate": 9.859738533523383e-05,
+      "loss": 1.155,
+      "num_input_tokens_seen": 1751904,
+      "step": 95
+    },
+    {
+      "epoch": 0.38341720584711236,
+      "grad_norm": 1.6448779106140137,
+      "learning_rate": 9.833719607645324e-05,
+      "loss": 1.0645,
+      "num_input_tokens_seen": 1844404,
+      "step": 100
+    },
+    {
+      "epoch": 0.38341720584711236,
+      "eval_loss": 0.9265391826629639,
+      "eval_runtime": 0.8899,
+      "eval_samples_per_second": 168.564,
+      "eval_steps_per_second": 42.703,
+      "num_input_tokens_seen": 1844404,
+      "step": 100
+    },
+    {
+      "epoch": 0.402588066139468,
+      "grad_norm": 3.241382598876953,
+      "learning_rate": 9.805528638214542e-05,
+      "loss": 1.1597,
+      "num_input_tokens_seen": 1930376,
+      "step": 105
+    },
+    {
+      "epoch": 0.4217589264318236,
+      "grad_norm": 1.7404630184173584,
+      "learning_rate": 9.77517829291108e-05,
+      "loss": 1.112,
+      "num_input_tokens_seen": 2014196,
+      "step": 110
+    },
+    {
+      "epoch": 0.44092978672417926,
+      "grad_norm": 2.0619559288024902,
+      "learning_rate": 9.742682209735727e-05,
+      "loss": 1.1078,
+      "num_input_tokens_seen": 2105080,
+      "step": 115
+    },
+    {
+      "epoch": 0.46010064701653486,
+      "grad_norm": 1.618997573852539,
+      "learning_rate": 9.708054990881763e-05,
+      "loss": 1.123,
+      "num_input_tokens_seen": 2212296,
+      "step": 120
+    },
+    {
+      "epoch": 0.4792715073088905,
+      "grad_norm": 2.0891501903533936,
+      "learning_rate": 9.671312196173412e-05,
+      "loss": 1.1139,
+      "num_input_tokens_seen": 2297740,
+      "step": 125
+    },
+    {
+      "epoch": 0.4984423676012461,
+      "grad_norm": 2.8104231357574463,
+      "learning_rate": 9.632470336074009e-05,
+      "loss": 1.1419,
+      "num_input_tokens_seen": 2393552,
+      "step": 130
+    },
+    {
+      "epoch": 0.5176132278936018,
+      "grad_norm": 2.313002586364746,
+      "learning_rate": 9.591546864266983e-05,
+      "loss": 1.1122,
+      "num_input_tokens_seen": 2485636,
+      "step": 135
+    },
+    {
+      "epoch": 0.5367840881859574,
+      "grad_norm": 1.6471989154815674,
+      "learning_rate": 9.548560169812997e-05,
+      "loss": 1.0872,
+      "num_input_tokens_seen": 2579380,
+      "step": 140
+    },
+    {
+      "epoch": 0.555954948478313,
+      "grad_norm": 1.8052936792373657,
+      "learning_rate": 9.50352956888678e-05,
+      "loss": 1.1084,
+      "num_input_tokens_seen": 2660580,
+      "step": 145
+    },
+    {
+      "epoch": 0.5751258087706685,
+      "grad_norm": 1.9831300973892212,
+      "learning_rate": 9.45647529609736e-05,
+      "loss": 1.0753,
+      "num_input_tokens_seen": 2752664,
+      "step": 150
+    },
+    {
+      "epoch": 0.5942966690630243,
+      "grad_norm": 1.9232532978057861,
+      "learning_rate": 9.4074184953956e-05,
+      "loss": 1.0932,
+      "num_input_tokens_seen": 2835612,
+      "step": 155
+    },
+    {
+      "epoch": 0.6134675293553798,
+      "grad_norm": 3.607057571411133,
+      "learning_rate": 9.356381210573091e-05,
+      "loss": 1.0856,
+      "num_input_tokens_seen": 2927884,
+      "step": 160
+    },
+    {
+      "epoch": 0.6326383896477354,
+      "grad_norm": 1.5196906328201294,
+      "learning_rate": 9.303386375356752e-05,
+      "loss": 1.0765,
+      "num_input_tokens_seen": 3011316,
+      "step": 165
+    },
+    {
+      "epoch": 0.651809249940091,
+      "grad_norm": 2.0994279384613037,
+      "learning_rate": 9.248457803103476e-05,
+      "loss": 1.0743,
+      "num_input_tokens_seen": 3113324,
+      "step": 170
+    },
+    {
+      "epoch": 0.6709801102324466,
+      "grad_norm": 2.0873262882232666,
+      "learning_rate": 9.191620176099558e-05,
+      "loss": 1.0799,
+      "num_input_tokens_seen": 3205472,
+      "step": 175
+    },
+    {
+      "epoch": 0.6901509705248023,
+      "grad_norm": 1.7372593879699707,
+      "learning_rate": 9.132899034469647e-05,
+      "loss": 1.095,
+      "num_input_tokens_seen": 3303772,
+      "step": 180
+    },
+    {
+      "epoch": 0.7093218308171579,
+      "grad_norm": 1.7090203762054443,
+      "learning_rate": 9.072320764700223e-05,
+      "loss": 1.0832,
+      "num_input_tokens_seen": 3386080,
+      "step": 185
+    },
+    {
+      "epoch": 0.7284926911095135,
+      "grad_norm": 1.707291603088379,
+      "learning_rate": 9.009912587782771e-05,
+      "loss": 1.0376,
+      "num_input_tokens_seen": 3471524,
+      "step": 190
+    },
+    {
+      "epoch": 0.7476635514018691,
+      "grad_norm": 1.663960576057434,
+      "learning_rate": 8.945702546981969e-05,
+      "loss": 1.0802,
+      "num_input_tokens_seen": 3556336,
+      "step": 195
+    },
+    {
+      "epoch": 0.7668344116942247,
+      "grad_norm": 1.8464930057525635,
+      "learning_rate": 8.879719495234363e-05,
+      "loss": 1.0769,
+      "num_input_tokens_seen": 3640408,
+      "step": 200
+    },
+    {
+      "epoch": 0.7668344116942247,
+      "eval_loss": 0.862065851688385,
+      "eval_runtime": 0.8351,
+      "eval_samples_per_second": 179.628,
+      "eval_steps_per_second": 45.506,
+      "num_input_tokens_seen": 3640408,
+      "step": 200
+    },
+    {
+      "epoch": 0.7860052719865804,
+      "grad_norm": 1.7220999002456665,
+      "learning_rate": 8.811993082183243e-05,
+      "loss": 1.078,
+      "num_input_tokens_seen": 3731324,
+      "step": 205
+    },
+    {
+      "epoch": 0.805176132278936,
+      "grad_norm": 1.7034516334533691,
+      "learning_rate": 8.742553740855506e-05,
+      "loss": 1.0565,
+      "num_input_tokens_seen": 3822944,
+      "step": 210
+    },
+    {
+      "epoch": 0.8243469925712916,
+      "grad_norm": 2.327296257019043,
+      "learning_rate": 8.671432673986494e-05,
+      "loss": 1.0721,
+      "num_input_tokens_seen": 3922476,
+      "step": 215
+    },
+    {
+      "epoch": 0.8435178528636472,
+      "grad_norm": 1.6464370489120483,
+      "learning_rate": 8.598661839998972e-05,
+      "loss": 1.0573,
+      "num_input_tokens_seen": 4003388,
+      "step": 220
+    },
+    {
+      "epoch": 0.8626887131560029,
+      "grad_norm": 2.116698741912842,
+      "learning_rate": 8.524273938642538e-05,
+      "loss": 1.0459,
+      "num_input_tokens_seen": 4084052,
+      "step": 225
+    },
+    {
+      "epoch": 0.8818595734483585,
+      "grad_norm": 1.5513197183609009,
+      "learning_rate": 8.448302396299905e-05,
+      "loss": 1.073,
+      "num_input_tokens_seen": 4177072,
+      "step": 230
+    },
+    {
+      "epoch": 0.9010304337407141,
+      "grad_norm": 1.8118634223937988,
+      "learning_rate": 8.370781350966683e-05,
+      "loss": 1.0786,
+      "num_input_tokens_seen": 4272004,
+      "step": 235
+    },
+    {
+      "epoch": 0.9202012940330697,
+      "grad_norm": 1.542823076248169,
+      "learning_rate": 8.291745636911382e-05,
+      "loss": 1.0556,
+      "num_input_tokens_seen": 4367124,
+      "step": 240
+    },
+    {
+      "epoch": 0.9393721543254253,
+      "grad_norm": 1.5214637517929077,
+      "learning_rate": 8.211230769022551e-05,
+      "loss": 1.052,
+      "num_input_tokens_seen": 4454860,
+      "step": 245
+    },
+    {
+      "epoch": 0.958543014617781,
+      "grad_norm": 1.5787030458450317,
+      "learning_rate": 8.129272926850079e-05,
+      "loss": 1.032,
+      "num_input_tokens_seen": 4544744,
+      "step": 250
+    },
+    {
+      "epoch": 0.9777138749101366,
+      "grad_norm": 1.842089295387268,
+      "learning_rate": 8.045908938347828e-05,
+      "loss": 1.0585,
+      "num_input_tokens_seen": 4627284,
+      "step": 255
+    },
+    {
+      "epoch": 0.9968847352024922,
+      "grad_norm": 1.8476406335830688,
+      "learning_rate": 7.961176263324901e-05,
+      "loss": 1.0045,
+      "num_input_tokens_seen": 4715336,
+      "step": 260
+    },
+    {
+      "epoch": 1.0160555954948478,
+      "grad_norm": 1.540380835533142,
+      "learning_rate": 7.875112976612984e-05,
+      "loss": 0.8789,
+      "num_input_tokens_seen": 4811720,
+      "step": 265
+    },
+    {
+      "epoch": 1.0352264557872035,
+      "grad_norm": 1.593477725982666,
+      "learning_rate": 7.787757750957334e-05,
+      "loss": 0.8548,
+      "num_input_tokens_seen": 4904124,
+      "step": 270
+    },
+    {
+      "epoch": 1.054397316079559,
+      "grad_norm": 2.4607889652252197,
+      "learning_rate": 7.699149839639086e-05,
+      "loss": 0.8393,
+      "num_input_tokens_seen": 4997508,
+      "step": 275
+    },
+    {
+      "epoch": 1.0735681763719147,
+      "grad_norm": 2.0560553073883057,
+      "learning_rate": 7.609329058836695e-05,
+      "loss": 0.8517,
+      "num_input_tokens_seen": 5102324,
+      "step": 280
+    },
+    {
+      "epoch": 1.0927390366642702,
+      "grad_norm": 1.5116240978240967,
+      "learning_rate": 7.518335769734439e-05,
+      "loss": 0.8498,
+      "num_input_tokens_seen": 5203192,
+      "step": 285
+    },
+    {
+      "epoch": 1.111909896956626,
+      "grad_norm": 1.517767071723938,
+      "learning_rate": 7.426210860386031e-05,
+      "loss": 0.8269,
+      "num_input_tokens_seen": 5300184,
+      "step": 290
+    },
+    {
+      "epoch": 1.1310807572489816,
+      "grad_norm": 1.6254216432571411,
+      "learning_rate": 7.332995727341462e-05,
+      "loss": 0.8496,
+      "num_input_tokens_seen": 5405724,
+      "step": 295
+    },
+    {
+      "epoch": 1.150251617541337,
+      "grad_norm": 1.6421043872833252,
+      "learning_rate": 7.238732257045372e-05,
+      "loss": 0.849,
+      "num_input_tokens_seen": 5504644,
+      "step": 300
+    },
+    {
+      "epoch": 1.150251617541337,
+      "eval_loss": 0.8501759767532349,
+      "eval_runtime": 0.8525,
+      "eval_samples_per_second": 175.946,
+      "eval_steps_per_second": 44.573,
+      "num_input_tokens_seen": 5504644,
+      "step": 300
+    },
+    {
+      "epoch": 1.1694224778336928,
+      "grad_norm": 1.7481452226638794,
+      "learning_rate": 7.143462807015271e-05,
+      "loss": 0.8823,
+      "num_input_tokens_seen": 5597560,
+      "step": 305
+    },
+    {
+      "epoch": 1.1885933381260485,
+      "grad_norm": 1.725755214691162,
+      "learning_rate": 7.047230186808085e-05,
+      "loss": 0.8701,
+      "num_input_tokens_seen": 5691244,
+      "step": 310
+    },
+    {
+      "epoch": 1.207764198418404,
+      "grad_norm": 1.4096667766571045,
+      "learning_rate": 6.950077638783578e-05,
+      "loss": 0.8501,
+      "num_input_tokens_seen": 5773336,
+      "step": 315
+    },
+    {
+      "epoch": 1.2269350587107597,
+      "grad_norm": 1.5906323194503784,
+      "learning_rate": 6.8520488186733e-05,
+      "loss": 0.8684,
+      "num_input_tokens_seen": 5849224,
+      "step": 320
+    },
+    {
+      "epoch": 1.2461059190031152,
+      "grad_norm": 2.2204673290252686,
+      "learning_rate": 6.753187775963773e-05,
+      "loss": 0.8688,
+      "num_input_tokens_seen": 5943104,
+      "step": 325
+    },
+    {
+      "epoch": 1.2652767792954709,
+      "grad_norm": 1.40131413936615,
+      "learning_rate": 6.653538934102743e-05,
+      "loss": 0.8495,
+      "num_input_tokens_seen": 6051232,
+      "step": 330
+    },
+    {
+      "epoch": 1.2844476395878264,
+      "grad_norm": 1.578782558441162,
+      "learning_rate": 6.553147070537413e-05,
+      "loss": 0.8218,
+      "num_input_tokens_seen": 6137520,
+      "step": 335
+    },
+    {
+      "epoch": 1.303618499880182,
+      "grad_norm": 1.5068399906158447,
+      "learning_rate": 6.452057296593568e-05,
+      "loss": 0.8964,
+      "num_input_tokens_seen": 6237012,
+      "step": 340
+    },
+    {
+      "epoch": 1.3227893601725378,
+      "grad_norm": 1.6630327701568604,
+      "learning_rate": 6.350315037204714e-05,
+      "loss": 0.8748,
+      "num_input_tokens_seen": 6332764,
+      "step": 345
+    },
+    {
+      "epoch": 1.3419602204648933,
+      "grad_norm": 1.569263219833374,
+      "learning_rate": 6.247966010500258e-05,
+      "loss": 0.8478,
+      "num_input_tokens_seen": 6416688,
+      "step": 350
+    },
+    {
+      "epoch": 1.361131080757249,
+      "grad_norm": 1.4157441854476929,
+      "learning_rate": 6.145056207261964e-05,
+      "loss": 0.8624,
+      "num_input_tokens_seen": 6507660,
+      "step": 355
+    },
+    {
+      "epoch": 1.3803019410496047,
+      "grad_norm": 1.4510629177093506,
+      "learning_rate": 6.0416318702578826e-05,
+      "loss": 0.851,
+      "num_input_tokens_seen": 6608708,
+      "step": 360
+    },
+    {
+      "epoch": 1.3994728013419602,
+      "grad_norm": 1.660876989364624,
+      "learning_rate": 5.9377394734630464e-05,
+      "loss": 0.8401,
+      "num_input_tokens_seen": 6700852,
+      "step": 365
+    },
+    {
+      "epoch": 1.4186436616343159,
+      "grad_norm": 1.4707056283950806,
+      "learning_rate": 5.833425701176294e-05,
+      "loss": 0.8646,
+      "num_input_tokens_seen": 6792244,
+      "step": 370
+    },
+    {
+      "epoch": 1.4378145219266716,
+      "grad_norm": 1.563291311264038,
+      "learning_rate": 5.728737427042548e-05,
+      "loss": 0.8732,
+      "num_input_tokens_seen": 6875536,
+      "step": 375
+    },
+    {
+      "epoch": 1.456985382219027,
+      "grad_norm": 1.286574125289917,
+      "learning_rate": 5.623721692990039e-05,
+      "loss": 0.8449,
+      "num_input_tokens_seen": 6958384,
+      "step": 380
+    },
+    {
+      "epoch": 1.4761562425113828,
+      "grad_norm": 1.413732886314392,
+      "learning_rate": 5.518425688091906e-05,
+      "loss": 0.8459,
+      "num_input_tokens_seen": 7040740,
+      "step": 385
+    },
+    {
+      "epoch": 1.4953271028037383,
+      "grad_norm": 1.3040345907211304,
+      "learning_rate": 5.4128967273616625e-05,
+      "loss": 0.8539,
+      "num_input_tokens_seen": 7132892,
+      "step": 390
+    },
+    {
+      "epoch": 1.514497963096094,
+      "grad_norm": 1.4091562032699585,
+      "learning_rate": 5.307182230492088e-05,
+      "loss": 0.816,
+      "num_input_tokens_seen": 7217480,
+      "step": 395
+    },
+    {
+      "epoch": 1.5336688233884495,
+      "grad_norm": 1.5895339250564575,
+      "learning_rate": 5.201329700547076e-05,
+      "loss": 0.8612,
+      "num_input_tokens_seen": 7316212,
+      "step": 400
+    },
+    {
+      "epoch": 1.5336688233884495,
+      "eval_loss": 0.8288899064064026,
+      "eval_runtime": 0.8525,
+      "eval_samples_per_second": 175.948,
+      "eval_steps_per_second": 44.574,
+      "num_input_tokens_seen": 7316212,
+      "step": 400
+    },
+    {
+      "epoch": 1.5528396836808052,
+      "grad_norm": 1.523808240890503,
+      "learning_rate": 5.095386702616012e-05,
+      "loss": 0.8411,
+      "num_input_tokens_seen": 7397436,
+      "step": 405
+    },
+    {
+      "epoch": 1.5720105439731609,
+      "grad_norm": 1.4366319179534912,
+      "learning_rate": 4.989400842440289e-05,
+      "loss": 0.8179,
+      "num_input_tokens_seen": 7489804,
+      "step": 410
+    },
+    {
+      "epoch": 1.5911814042655164,
+      "grad_norm": 1.335564136505127,
+      "learning_rate": 4.883419745021554e-05,
+      "loss": 0.8321,
+      "num_input_tokens_seen": 7575200,
+      "step": 415
+    },
+    {
+      "epoch": 1.610352264557872,
+      "grad_norm": 1.5225696563720703,
+      "learning_rate": 4.7774910332213e-05,
+      "loss": 0.8408,
+      "num_input_tokens_seen": 7661620,
+      "step": 420
+    },
+    {
+      "epoch": 1.6295231248502278,
+      "grad_norm": 1.286909580230713,
+      "learning_rate": 4.6716623063614094e-05,
+      "loss": 0.8335,
+      "num_input_tokens_seen": 7751008,
+      "step": 425
+    },
+    {
+      "epoch": 1.6486939851425833,
+      "grad_norm": 1.399095058441162,
+      "learning_rate": 4.565981118835299e-05,
+      "loss": 0.8586,
+      "num_input_tokens_seen": 7847504,
+      "step": 430
+    },
+    {
+      "epoch": 1.6678648454349387,
+      "grad_norm": 1.5877128839492798,
+      "learning_rate": 4.4604949587392234e-05,
+      "loss": 0.8451,
+      "num_input_tokens_seen": 7940004,
+      "step": 435
+    },
+    {
+      "epoch": 1.6870357057272947,
+      "grad_norm": 1.661634087562561,
+      "learning_rate": 4.355251226533396e-05,
+      "loss": 0.8517,
+      "num_input_tokens_seen": 8037312,
+      "step": 440
+    },
+    {
+      "epoch": 1.7062065660196502,
+      "grad_norm": 1.2558486461639404,
+      "learning_rate": 4.250297213742473e-05,
+      "loss": 0.8115,
+      "num_input_tokens_seen": 8134436,
+      "step": 445
+    },
+    {
+      "epoch": 1.7253774263120056,
+      "grad_norm": 1.4006855487823486,
+      "learning_rate": 4.145680081704989e-05,
+      "loss": 0.8471,
+      "num_input_tokens_seen": 8223592,
+      "step": 450
+    },
+    {
+      "epoch": 1.7445482866043613,
+      "grad_norm": 1.5870461463928223,
+      "learning_rate": 4.0414468403813095e-05,
+      "loss": 0.8582,
+      "num_input_tokens_seen": 8312984,
+      "step": 455
+    },
+    {
+      "epoch": 1.763719146896717,
+      "grad_norm": 1.4359045028686523,
+      "learning_rate": 3.937644327229572e-05,
+      "loss": 0.8146,
+      "num_input_tokens_seen": 8402888,
+      "step": 460
+    },
+    {
+      "epoch": 1.7828900071890725,
+      "grad_norm": 1.4591436386108398,
+      "learning_rate": 3.8343191861591795e-05,
+      "loss": 0.8276,
+      "num_input_tokens_seen": 8503552,
+      "step": 465
+    },
+    {
+      "epoch": 1.8020608674814282,
+      "grad_norm": 1.6432543992996216,
+      "learning_rate": 3.7315178465712366e-05,
+      "loss": 0.8524,
+      "num_input_tokens_seen": 8604500,
+      "step": 470
+    },
+    {
+      "epoch": 1.821231727773784,
+      "grad_norm": 1.58722984790802,
+      "learning_rate": 3.629286502495394e-05,
+      "loss": 0.8257,
+      "num_input_tokens_seen": 8690612,
+      "step": 475
+    },
+    {
+      "epoch": 1.8404025880661394,
+      "grad_norm": 1.3321154117584229,
+      "learning_rate": 3.52767109183244e-05,
+      "loss": 0.8194,
+      "num_input_tokens_seen": 8785132,
+      "step": 480
+    },
+    {
+      "epoch": 1.8595734483584951,
+      "grad_norm": 1.8727712631225586,
+      "learning_rate": 3.426717275712e-05,
+      "loss": 0.8414,
+      "num_input_tokens_seen": 8882700,
+      "step": 485
+    },
+    {
+      "epoch": 1.8787443086508508,
+      "grad_norm": 1.427962303161621,
+      "learning_rate": 3.326470417974604e-05,
+      "loss": 0.8275,
+      "num_input_tokens_seen": 8982636,
+      "step": 490
+    },
+    {
+      "epoch": 1.8979151689432063,
+      "grad_norm": 1.391340970993042,
+      "learning_rate": 3.226975564787322e-05,
+      "loss": 0.8115,
+      "num_input_tokens_seen": 9090140,
+      "step": 495
+    },
+    {
+      "epoch": 1.9170860292355618,
+      "grad_norm": 1.465030312538147,
+      "learning_rate": 3.1282774244021715e-05,
+      "loss": 0.7934,
+      "num_input_tokens_seen": 9167936,
+      "step": 500
+    },
+    {
+      "epoch": 1.9170860292355618,
+      "eval_loss": 0.8071622252464294,
+      "eval_runtime": 0.8763,
+      "eval_samples_per_second": 171.177,
+      "eval_steps_per_second": 43.365,
+      "num_input_tokens_seen": 9167936,
+      "step": 500
+    },
+    {
+      "epoch": 1.9362568895279175,
+      "grad_norm": 1.6803951263427734,
+      "learning_rate": 3.0304203470663505e-05,
+      "loss": 0.821,
+      "num_input_tokens_seen": 9252204,
+      "step": 505
+    },
+    {
+      "epoch": 1.9554277498202732,
+      "grad_norm": 1.4982653856277466,
+      "learning_rate": 2.9334483050933503e-05,
+      "loss": 0.7982,
+      "num_input_tokens_seen": 9326444,
+      "step": 510
+    },
+    {
+      "epoch": 1.9745986101126287,
+      "grad_norm": 1.5553169250488281,
+      "learning_rate": 2.8374048731038898e-05,
+      "loss": 0.8183,
+      "num_input_tokens_seen": 9412396,
+      "step": 515
+    },
+    {
+      "epoch": 1.9937694704049844,
+      "grad_norm": 1.176651954650879,
+      "learning_rate": 2.7423332084455544e-05,
+      "loss": 0.7837,
+      "num_input_tokens_seen": 9497412,
+      "step": 520
+    },
+    {
+      "epoch": 2.01294033069734,
+      "grad_norm": 1.2269119024276733,
+      "learning_rate": 2.648276031799934e-05,
+      "loss": 0.726,
+      "num_input_tokens_seen": 9594720,
+      "step": 525
+    },
+    {
+      "epoch": 2.0321111909896956,
+      "grad_norm": 1.4332973957061768,
+      "learning_rate": 2.5552756079859903e-05,
+      "loss": 0.6847,
+      "num_input_tokens_seen": 9675564,
+      "step": 530
+    },
+    {
+      "epoch": 2.051282051282051,
+      "grad_norm": 1.3018600940704346,
+      "learning_rate": 2.4633737269682543e-05,
+      "loss": 0.682,
+      "num_input_tokens_seen": 9772140,
+      "step": 535
+    },
+    {
+      "epoch": 2.070452911574407,
+      "grad_norm": 1.2701021432876587,
+      "learning_rate": 2.3726116850783985e-05,
+      "loss": 0.6643,
+      "num_input_tokens_seen": 9850508,
+      "step": 540
+    },
+    {
+      "epoch": 2.0896237718667625,
+      "grad_norm": 1.2966877222061157,
+      "learning_rate": 2.283030266458644e-05,
+      "loss": 0.6868,
+      "num_input_tokens_seen": 9949592,
+      "step": 545
+    },
+    {
+      "epoch": 2.108794632159118,
+      "grad_norm": 1.4499340057373047,
+      "learning_rate": 2.194669724735296e-05,
+      "loss": 0.6713,
+      "num_input_tokens_seen": 10041244,
+      "step": 550
+    },
+    {
+      "epoch": 2.127965492451474,
+      "grad_norm": 1.791567325592041,
+      "learning_rate": 2.1075697649306835e-05,
+      "loss": 0.6893,
+      "num_input_tokens_seen": 10140004,
+      "step": 555
+    },
+    {
+      "epoch": 2.1471363527438294,
+      "grad_norm": 1.4607737064361572,
+      "learning_rate": 2.0217695256216195e-05,
+      "loss": 0.6784,
+      "num_input_tokens_seen": 10233820,
+      "step": 560
+    },
+    {
+      "epoch": 2.166307213036185,
+      "grad_norm": 1.4586418867111206,
+      "learning_rate": 1.937307561352373e-05,
+      "loss": 0.6829,
+      "num_input_tokens_seen": 10324844,
+      "step": 565
+    },
+    {
+      "epoch": 2.1854780733285404,
+      "grad_norm": 1.4719526767730713,
+      "learning_rate": 1.854221825310103e-05,
+      "loss": 0.6775,
+      "num_input_tokens_seen": 10413592,
+      "step": 570
+    },
+    {
+      "epoch": 2.2046489336208963,
+      "grad_norm": 1.4231044054031372,
+      "learning_rate": 1.7725496522704998e-05,
+      "loss": 0.6872,
+      "num_input_tokens_seen": 10503432,
+      "step": 575
+    },
+    {
+      "epoch": 2.223819793913252,
+      "grad_norm": 1.3985334634780884,
+      "learning_rate": 1.6923277418213117e-05,
+      "loss": 0.65,
+      "num_input_tokens_seen": 10600988,
+      "step": 580
+    },
+    {
+      "epoch": 2.2429906542056073,
+      "grad_norm": 1.3148375749588013,
+      "learning_rate": 1.6135921418712956e-05,
+      "loss": 0.6901,
+      "num_input_tokens_seen": 10692664,
+      "step": 585
+    },
+    {
+      "epoch": 2.262161514497963,
+      "grad_norm": 1.3653095960617065,
+      "learning_rate": 1.536378232452003e-05,
+      "loss": 0.6661,
+      "num_input_tokens_seen": 10781516,
+      "step": 590
+    },
+    {
+      "epoch": 2.2813323747903187,
+      "grad_norm": 1.4331799745559692,
+      "learning_rate": 1.4607207098196852e-05,
+      "loss": 0.669,
+      "num_input_tokens_seen": 10874028,
+      "step": 595
+    },
+    {
+      "epoch": 2.300503235082674,
+      "grad_norm": 1.3715401887893677,
+      "learning_rate": 1.3866535708644334e-05,
+      "loss": 0.6701,
+      "num_input_tokens_seen": 10969348,
+      "step": 600
+    },
+    {
+      "epoch": 2.300503235082674,
+      "eval_loss": 0.8050560355186462,
+      "eval_runtime": 0.8838,
+      "eval_samples_per_second": 169.723,
+      "eval_steps_per_second": 42.997,
+      "num_input_tokens_seen": 10969348,
+      "step": 600
+    },
+    {
+      "epoch": 2.31967409537503,
+      "grad_norm": 1.2875298261642456,
+      "learning_rate": 1.3142100978336069e-05,
+      "loss": 0.6877,
+      "num_input_tokens_seen": 11064696,
+      "step": 605
+    },
+    {
+      "epoch": 2.3388449556673856,
+      "grad_norm": 1.2775810956954956,
+      "learning_rate": 1.2434228433763657e-05,
+      "loss": 0.6752,
+      "num_input_tokens_seen": 11157516,
+      "step": 610
+    },
+    {
+      "epoch": 2.358015815959741,
+      "grad_norm": 1.3270821571350098,
+      "learning_rate": 1.1743236159160653e-05,
+      "loss": 0.6915,
+      "num_input_tokens_seen": 11236376,
+      "step": 615
+    },
+    {
+      "epoch": 2.377186676252097,
+      "grad_norm": 1.2592318058013916,
+      "learning_rate": 1.1069434653570631e-05,
+      "loss": 0.6786,
+      "num_input_tokens_seen": 11343512,
+      "step": 620
+    },
+    {
+      "epoch": 2.3963575365444525,
+      "grad_norm": 1.4185535907745361,
+      "learning_rate": 1.0413126691323666e-05,
+      "loss": 0.6876,
+      "num_input_tokens_seen": 11433156,
+      "step": 625
+    },
+    {
+      "epoch": 2.415528396836808,
+      "grad_norm": 1.5168098211288452,
+      "learning_rate": 9.774607185984002e-06,
+      "loss": 0.6911,
+      "num_input_tokens_seen": 11517328,
+      "step": 630
+    },
+    {
+      "epoch": 2.4346992571291635,
+      "grad_norm": 1.2858244180679321,
+      "learning_rate": 9.154163057829879e-06,
+      "loss": 0.6618,
+      "num_input_tokens_seen": 11602144,
+      "step": 635
+    },
+    {
+      "epoch": 2.4538701174215194,
+      "grad_norm": 1.2297165393829346,
+      "learning_rate": 8.552073104925295e-06,
+      "loss": 0.6804,
+      "num_input_tokens_seen": 11691216,
+      "step": 640
+    },
+    {
+      "epoch": 2.473040977713875,
+      "grad_norm": 1.2545212507247925,
+      "learning_rate": 7.968607877841332e-06,
+      "loss": 0.6669,
+      "num_input_tokens_seen": 11789340,
+      "step": 645
+    },
+    {
+      "epoch": 2.4922118380062304,
+      "grad_norm": 1.3260319232940674,
+      "learning_rate": 7.404029558083653e-06,
+      "loss": 0.6847,
+      "num_input_tokens_seen": 11884208,
+      "step": 650
+    },
+    {
+      "epoch": 2.5113826982985863,
+      "grad_norm": 1.2979117631912231,
+      "learning_rate": 6.858591840280626e-06,
+      "loss": 0.6592,
+      "num_input_tokens_seen": 11972124,
+      "step": 655
+    },
+    {
+      "epoch": 2.5305535585909418,
+      "grad_norm": 1.2984734773635864,
+      "learning_rate": 6.3325398181849845e-06,
+      "loss": 0.6579,
+      "num_input_tokens_seen": 12057320,
+      "step": 660
+    },
+    {
+      "epoch": 2.5497244188832973,
+      "grad_norm": 1.2366037368774414,
+      "learning_rate": 5.826109874540409e-06,
+      "loss": 0.6666,
+      "num_input_tokens_seen": 12154952,
+      "step": 665
+    },
+    {
+      "epoch": 2.5688952791756527,
+      "grad_norm": 1.2756296396255493,
+      "learning_rate": 5.33952957486234e-06,
+      "loss": 0.6903,
+      "num_input_tokens_seen": 12256604,
+      "step": 670
+    },
+    {
+      "epoch": 2.5880661394680087,
+      "grad_norm": 1.3016470670700073,
+      "learning_rate": 4.873017565180871e-06,
+      "loss": 0.6654,
+      "num_input_tokens_seen": 12341988,
+      "step": 675
+    },
+    {
+      "epoch": 2.607236999760364,
+      "grad_norm": 1.3244975805282593,
+      "learning_rate": 4.4267834737916296e-06,
+      "loss": 0.6597,
+      "num_input_tokens_seen": 12442644,
+      "step": 680
+    },
+    {
+      "epoch": 2.62640786005272,
+      "grad_norm": 1.4668656587600708,
+      "learning_rate": 4.001027817058789e-06,
+      "loss": 0.6709,
+      "num_input_tokens_seen": 12538688,
+      "step": 685
+    },
+    {
+      "epoch": 2.6455787203450756,
+      "grad_norm": 1.4666439294815063,
+      "learning_rate": 3.5959419093125946e-06,
+      "loss": 0.6793,
+      "num_input_tokens_seen": 12621632,
+      "step": 690
+    },
+    {
+      "epoch": 2.664749580637431,
+      "grad_norm": 1.2680643796920776,
+      "learning_rate": 3.211707776881739e-06,
+      "loss": 0.6562,
+      "num_input_tokens_seen": 12715832,
+      "step": 695
+    },
+    {
+      "epoch": 2.6839204409297865,
+      "grad_norm": 1.3070735931396484,
+      "learning_rate": 2.848498076299483e-06,
+      "loss": 0.6579,
+      "num_input_tokens_seen": 12814620,
+      "step": 700
+    },
+    {
+      "epoch": 2.6839204409297865,
+      "eval_loss": 0.7903470396995544,
+      "eval_runtime": 0.941,
+      "eval_samples_per_second": 159.398,
+      "eval_steps_per_second": 40.381,
+      "num_input_tokens_seen": 12814620,
+      "step": 700
+    },
+    {
+      "epoch": 2.7030913012221425,
+      "grad_norm": 1.4454238414764404,
+      "learning_rate": 2.506476016719922e-06,
+      "loss": 0.6818,
+      "num_input_tokens_seen": 12909896,
+      "step": 705
+    },
+    {
+      "epoch": 2.722262161514498,
+      "grad_norm": 1.249842882156372,
+      "learning_rate": 2.1857952865796614e-06,
+      "loss": 0.6655,
+      "num_input_tokens_seen": 13001288,
+      "step": 710
+    },
+    {
+      "epoch": 2.7414330218068534,
+      "grad_norm": 1.312296986579895,
+      "learning_rate": 1.8865999845374793e-06,
+      "loss": 0.656,
+      "num_input_tokens_seen": 13094568,
+      "step": 715
+    },
+    {
+      "epoch": 2.7606038820992094,
+      "grad_norm": 1.1614197492599487,
+      "learning_rate": 1.6090245547232707e-06,
+      "loss": 0.6664,
+      "num_input_tokens_seen": 13181872,
+      "step": 720
+    },
+    {
+      "epoch": 2.779774742391565,
+      "grad_norm": 1.249230146408081,
+      "learning_rate": 1.353193726325247e-06,
+      "loss": 0.6643,
+      "num_input_tokens_seen": 13276044,
+      "step": 725
+    },
+    {
+      "epoch": 2.7989456026839203,
+      "grad_norm": 1.3258079290390015,
+      "learning_rate": 1.1192224575425848e-06,
+      "loss": 0.6352,
+      "num_input_tokens_seen": 13361008,
+      "step": 730
+    },
+    {
+      "epoch": 2.818116462976276,
+      "grad_norm": 1.364700198173523,
+      "learning_rate": 9.072158839286748e-07,
+      "loss": 0.6686,
+      "num_input_tokens_seen": 13458560,
+      "step": 735
+    },
+    {
+      "epoch": 2.8372873232686318,
+      "grad_norm": 1.3140686750411987,
+      "learning_rate": 7.172692711482021e-07,
+      "loss": 0.6623,
+      "num_input_tokens_seen": 13551928,
+      "step": 740
+    },
+    {
+      "epoch": 2.8564581835609872,
+      "grad_norm": 1.4460844993591309,
+      "learning_rate": 5.494679721693152e-07,
+      "loss": 0.6658,
+      "num_input_tokens_seen": 13632792,
+      "step": 745
+    },
+    {
+      "epoch": 2.875629043853343,
+      "grad_norm": 1.2111197710037231,
+      "learning_rate": 4.0388738891002366e-07,
+      "loss": 0.6239,
+      "num_input_tokens_seen": 13723172,
+      "step": 750
+    },
+    {
+      "epoch": 2.8947999041456987,
+      "grad_norm": 1.2612218856811523,
+      "learning_rate": 2.8059293835620003e-07,
+      "loss": 0.6682,
+      "num_input_tokens_seen": 13820332,
+      "step": 755
+    },
+    {
+      "epoch": 2.913970764438054,
+      "grad_norm": 1.3601187467575073,
+      "learning_rate": 1.7964002316628315e-07,
+      "loss": 0.6729,
+      "num_input_tokens_seen": 13907996,
+      "step": 760
+    },
+    {
+      "epoch": 2.9331416247304096,
+      "grad_norm": 1.3947527408599854,
+      "learning_rate": 1.0107400677596412e-07,
+      "loss": 0.6736,
+      "num_input_tokens_seen": 13993196,
+      "step": 765
+    },
+    {
+      "epoch": 2.9523124850227656,
+      "grad_norm": 1.2355984449386597,
+      "learning_rate": 4.493019301401446e-08,
+      "loss": 0.6655,
+      "num_input_tokens_seen": 14084984,
+      "step": 770
+    },
+    {
+      "epoch": 2.971483345315121,
+      "grad_norm": 1.2893033027648926,
+      "learning_rate": 1.1233810238425735e-08,
+      "loss": 0.657,
+      "num_input_tokens_seen": 14169848,
+      "step": 775
+    },
+    {
+      "epoch": 2.9906542056074765,
+      "grad_norm": 1.3213859796524048,
+      "learning_rate": 0.0,
+      "loss": 0.6589,
+      "num_input_tokens_seen": 14258488,
+      "step": 780
+    },
+    {
+      "epoch": 2.9906542056074765,
+      "num_input_tokens_seen": 14258488,
+      "step": 780,
+      "total_flos": 3.0175424769490944e+16,
+      "train_loss": 0.895275863011678,
+      "train_runtime": 860.318,
+      "train_samples_per_second": 58.206,
+      "train_steps_per_second": 0.907
+    }
+  ],
+  "logging_steps": 5,
+  "max_steps": 780,
+  "num_input_tokens_seen": 14258488,
+  "num_train_epochs": 3,
+  "save_steps": 100,
+  "stateful_callbacks": {
+    "TrainerControl": {
+      "args": {
+        "should_epoch_stop": false,
+        "should_evaluate": false,
+        "should_log": false,
+        "should_save": true,
+        "should_training_stop": true
+      },
+      "attributes": {}
+    }
+  },
+  "total_flos": 3.0175424769490944e+16,
+  "train_batch_size": 4,
+  "trial_name": null,
+  "trial_params": null
+}

training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4d85165a318ec430e48d0c7cff3da74e1488acdd1e9816f78db5817ca9366a4b
+size 5496