Upload 26 files

7b/iteration1/README.md

@@ -0,0 +1,202 @@
+---
+base_model: models/llava-1.5-7b-hf
+library_name: peft
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.14.0

7b/iteration1/adapter_config.json

@@ -0,0 +1,130 @@
+{
+  "alpha_pattern": {},
+  "auto_mapping": null,
+  "base_model_name_or_path": "models/llava-1.5-7b-hf",
+  "bias": "none",
+  "eva_config": null,
+  "exclude_modules": null,
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 256,
+  "lora_bias": false,
+  "lora_dropout": 0.05,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "r": 128,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "up_proj",
+    "24.self_attn.v_proj",
+    "29.self_attn.k_proj",
+    "26.self_attn.q_proj",
+    "model.layers.10.self_attn.v_proj",
+    "model.layers.7.self_attn.k_proj",
+    "25.self_attn.q_proj",
+    "model.layers.10.self_attn.k_proj",
+    "model.layers.12.self_attn.q_proj",
+    "model.layers.15.self_attn.k_proj",
+    "27.self_attn.q_proj",
+    "model.layers.13.self_attn.v_proj",
+    "model.layers.1.self_attn.v_proj",
+    "model.layers.20.self_attn.k_proj",
+    "model.layers.6.self_attn.k_proj",
+    "model.layers.23.self_attn.q_proj",
+    "model.layers.14.self_attn.v_proj",
+    "30.self_attn.q_proj",
+    "model.layers.20.self_attn.v_proj",
+    "31.self_attn.k_proj",
+    "model.layers.2.self_attn.k_proj",
+    "model.layers.6.self_attn.q_proj",
+    "model.layers.4.self_attn.q_proj",
+    "model.layers.17.self_attn.k_proj",
+    "model.layers.20.self_attn.q_proj",
+    "model.layers.4.self_attn.k_proj",
+    "28.self_attn.v_proj",
+    "30.self_attn.v_proj",
+    "model.layers.16.self_attn.k_proj",
+    "model.layers.8.self_attn.v_proj",
+    "model.layers.7.self_attn.v_proj",
+    "model.layers.1.self_attn.k_proj",
+    "model.layers.5.self_attn.q_proj",
+    "model.layers.3.self_attn.v_proj",
+    "model.layers.12.self_attn.v_proj",
+    "model.layers.21.self_attn.q_proj",
+    "model.layers.15.self_attn.v_proj",
+    "model.layers.22.self_attn.k_proj",
+    "31.self_attn.v_proj",
+    "29.self_attn.q_proj",
+    "gate_proj",
+    "model.layers.2.self_attn.q_proj",
+    "model.layers.11.self_attn.q_proj",
+    "model.layers.2.self_attn.v_proj",
+    "model.layers.23.self_attn.v_proj",
+    "model.layers.19.self_attn.k_proj",
+    "25.self_attn.k_proj",
+    "27.self_attn.v_proj",
+    "24.self_attn.q_proj",
+    "model.layers.13.self_attn.q_proj",
+    "31.self_attn.q_proj",
+    "model.layers.11.self_attn.k_proj",
+    "model.layers.0.self_attn.v_proj",
+    "model.layers.9.self_attn.k_proj",
+    "model.layers.0.self_attn.q_proj",
+    "model.layers.8.self_attn.k_proj",
+    "model.layers.17.self_attn.v_proj",
+    "model.layers.11.self_attn.v_proj",
+    "model.layers.0.self_attn.k_proj",
+    "model.layers.7.self_attn.q_proj",
+    "down_proj",
+    "25.self_attn.v_proj",
+    "29.self_attn.v_proj",
+    "model.layers.14.self_attn.q_proj",
+    "model.layers.17.self_attn.q_proj",
+    "model.layers.18.self_attn.v_proj",
+    "model.layers.12.self_attn.k_proj",
+    "model.layers.10.self_attn.q_proj",
+    "model.layers.5.self_attn.v_proj",
+    "model.layers.1.self_attn.q_proj",
+    "model.layers.18.self_attn.k_proj",
+    "30.self_attn.k_proj",
+    "26.self_attn.k_proj",
+    "model.layers.15.self_attn.q_proj",
+    "model.layers.19.self_attn.v_proj",
+    "model.layers.22.self_attn.v_proj",
+    "model.layers.3.self_attn.k_proj",
+    "model.layers.19.self_attn.q_proj",
+    "o_proj",
+    "model.layers.16.self_attn.q_proj",
+    "model.layers.9.self_attn.v_proj",
+    "model.layers.21.self_attn.k_proj",
+    "28.self_attn.q_proj",
+    "28.self_attn.k_proj",
+    "model.layers.13.self_attn.k_proj",
+    "24.self_attn.k_proj",
+    "model.layers.3.self_attn.q_proj",
+    "26.self_attn.v_proj",
+    "model.layers.16.self_attn.v_proj",
+    "model.layers.22.self_attn.q_proj",
+    "model.layers.18.self_attn.q_proj",
+    "27.self_attn.k_proj",
+    "model.layers.21.self_attn.v_proj",
+    "model.layers.9.self_attn.q_proj",
+    "model.layers.23.self_attn.k_proj",
+    "model.layers.5.self_attn.k_proj",
+    "model.layers.14.self_attn.k_proj",
+    "model.layers.6.self_attn.v_proj",
+    "model.layers.8.self_attn.q_proj",
+    "model.layers.4.self_attn.v_proj"
+  ],
+  "task_type": "CAUSAL_LM",
+  "use_dora": false,
+  "use_rslora": false
+}

7b/iteration1/adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:102168e24c5480c3d0e1aee6cf4c7dce7a9ea83b54acfb1a37be95f75eae093c
+size 639699488

7b/iteration1/added_tokens.json

@@ -0,0 +1,4 @@
+{
+  "<image>": 32000,
+  "<pad>": 32001
+}

7b/iteration1/chat_template.json

@@ -0,0 +1,3 @@
+{
+  "chat_template": "{% for message in messages %}{% if message['role'] != 'system' %}{{ message['role'].upper() + ': '}}{% endif %}{# Render all images first #}{% for content in message['content'] | selectattr('type', 'equalto', 'image') %}{{ '<image>\n' }}{% endfor %}{# Render all text next #}{% if message['role'] != 'assistant' %}{% for content in message['content'] | selectattr('type', 'equalto', 'text') %}{{ content['text'] + ' '}}{% endfor %}{% else %}{% for content in message['content'] | selectattr('type', 'equalto', 'text') %}{% generation %}{{ content['text'] + ' '}}{% endgeneration %}{% endfor %}{% endif %}{% endfor %}{% if add_generation_prompt %}{{ 'ASSISTANT:' }}{% endif %}"
+}

7b/iteration1/preprocessor_config.json

@@ -0,0 +1,28 @@
+{
+  "crop_size": {
+    "height": 336,
+    "width": 336
+  },
+  "do_center_crop": true,
+  "do_convert_rgb": true,
+  "do_normalize": true,
+  "do_rescale": true,
+  "do_resize": true,
+  "image_mean": [
+    0.48145466,
+    0.4578275,
+    0.40821073
+  ],
+  "image_processor_type": "CLIPImageProcessor",
+  "image_std": [
+    0.26862954,
+    0.26130258,
+    0.27577711
+  ],
+  "processor_class": "LlavaProcessor",
+  "resample": 3,
+  "rescale_factor": 0.00392156862745098,
+  "size": {
+    "shortest_edge": 336
+  }
+}

7b/iteration1/processor_config.json

@@ -0,0 +1,7 @@
+{
+  "image_token": "<image>",
+  "num_additional_image_tokens": 1,
+  "patch_size": 14,
+  "processor_class": "LlavaProcessor",
+  "vision_feature_select_strategy": "default"
+}

7b/iteration1/special_tokens_map.json

@@ -0,0 +1,31 @@
+{
+  "bos_token": {
+    "content": "<s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "</s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "image_token": "<image>",
+  "pad_token": {
+    "content": "<pad>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "unk_token": {
+    "content": "<unk>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

7b/iteration1/tokenizer.model

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9e556afd44213b6bd1be2b850ebbbd98f5481437a8021afaf58ee7fb1818d347
+size 499723

7b/iteration1/tokenizer_config.json

@@ -0,0 +1,64 @@
+{
+  "add_bos_token": true,
+  "add_eos_token": false,
+  "add_prefix_space": null,
+  "added_tokens_decoder": {
+    "0": {
+      "content": "<unk>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "1": {
+      "content": "<s>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "2": {
+      "content": "</s>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "32000": {
+      "content": "<image>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "32001": {
+      "content": "<pad>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "bos_token": "<s>",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "</s>",
+  "extra_special_tokens": {
+    "image_token": "<image>"
+  },
+  "image_token": "<image>",
+  "legacy": false,
+  "model_max_length": 1000000000000000019884624838656,
+  "pad_token": "<pad>",
+  "padding_side": "right",
+  "processor_class": "LlavaProcessor",
+  "sp_model_kwargs": {},
+  "tokenizer_class": "LlamaTokenizer",
+  "trust_remote_code": false,
+  "unk_token": "<unk>",
+  "use_default_system_prompt": false
+}

7b/iteration1/trainer_state.json

@@ -0,0 +1,2769 @@
+{
+  "best_metric": null,
+  "best_model_checkpoint": null,
+  "epoch": 1.0,
+  "eval_steps": 500,
+  "global_step": 171,
+  "is_hyper_param_search": false,
+  "is_local_process_zero": true,
+  "is_world_process_zero": true,
+  "log_history": [
+    {
+      "epoch": 0.005847953216374269,
+      "grad_norm": 12.401365280151367,
+      "learning_rate": 1.111111111111111e-06,
+      "logits/chosen": 0.130859375,
+      "logits/rejected": 0.173828125,
+      "logps/chosen": -110.5,
+      "logps/rejected": -13.0625,
+      "loss": 0.9935,
+      "nll_loss": 1.4140625,
+      "rewards/accuracies": 0.0,
+      "rewards/chosen": 0.0,
+      "rewards/margins": 0.0,
+      "rewards/rejected": 0.0,
+      "step": 1
+    },
+    {
+      "epoch": 0.011695906432748537,
+      "grad_norm": 14.383028984069824,
+      "learning_rate": 2.222222222222222e-06,
+      "logits/chosen": -0.0257568359375,
+      "logits/rejected": -0.41796875,
+      "logps/chosen": -121.0,
+      "logps/rejected": -59.5,
+      "loss": 0.9805,
+      "nll_loss": 1.3515625,
+      "rewards/accuracies": 0.0,
+      "rewards/chosen": 0.0,
+      "rewards/margins": 0.0,
+      "rewards/rejected": 0.0,
+      "step": 2
+    },
+    {
+      "epoch": 0.017543859649122806,
+      "grad_norm": 10.575273513793945,
+      "learning_rate": 3.3333333333333333e-06,
+      "logits/chosen": 0.1416015625,
+      "logits/rejected": 0.09521484375,
+      "logps/chosen": -106.0,
+      "logps/rejected": -37.5,
+      "loss": 1.0352,
+      "nll_loss": 1.5,
+      "rewards/accuracies": 0.25,
+      "rewards/chosen": 0.125,
+      "rewards/margins": 0.0859375,
+      "rewards/rejected": 0.0390625,
+      "step": 3
+    },
+    {
+      "epoch": 0.023391812865497075,
+      "grad_norm": 10.790839195251465,
+      "learning_rate": 4.444444444444444e-06,
+      "logits/chosen": 0.2119140625,
+      "logits/rejected": 0.25,
+      "logps/chosen": -76.0,
+      "logps/rejected": -7.0,
+      "loss": 0.9434,
+      "nll_loss": 1.46875,
+      "rewards/accuracies": 0.5,
+      "rewards/chosen": 0.0625,
+      "rewards/margins": 0.0703125,
+      "rewards/rejected": -0.0078125,
+      "step": 4
+    },
+    {
+      "epoch": 0.029239766081871343,
+      "grad_norm": 11.351024627685547,
+      "learning_rate": 5.555555555555557e-06,
+      "logits/chosen": 0.1611328125,
+      "logits/rejected": 0.1630859375,
+      "logps/chosen": -133.0,
+      "logps/rejected": -34.5,
+      "loss": 0.9577,
+      "nll_loss": 1.3984375,
+      "rewards/accuracies": 0.25,
+      "rewards/chosen": 0.0,
+      "rewards/margins": 0.03125,
+      "rewards/rejected": -0.03125,
+      "step": 5
+    },
+    {
+      "epoch": 0.03508771929824561,
+      "grad_norm": 10.957937240600586,
+      "learning_rate": 6.666666666666667e-06,
+      "logits/chosen": 0.193359375,
+      "logits/rejected": 0.2890625,
+      "logps/chosen": -72.0,
+      "logps/rejected": -16.5,
+      "loss": 0.9368,
+      "nll_loss": 1.4375,
+      "rewards/accuracies": 0.25,
+      "rewards/chosen": -0.015625,
+      "rewards/margins": 0.0,
+      "rewards/rejected": -0.015625,
+      "step": 6
+    },
+    {
+      "epoch": 0.04093567251461988,
+      "grad_norm": 10.016559600830078,
+      "learning_rate": 7.77777777777778e-06,
+      "logits/chosen": 0.474609375,
+      "logits/rejected": 0.287109375,
+      "logps/chosen": -50.5,
+      "logps/rejected": -17.875,
+      "loss": 0.8958,
+      "nll_loss": 1.5234375,
+      "rewards/accuracies": 0.75,
+      "rewards/chosen": 0.12109375,
+      "rewards/margins": 0.2109375,
+      "rewards/rejected": -0.08984375,
+      "step": 7
+    },
+    {
+      "epoch": 0.04678362573099415,
+      "grad_norm": 10.56991195678711,
+      "learning_rate": 8.888888888888888e-06,
+      "logits/chosen": -0.047119140625,
+      "logits/rejected": 0.205078125,
+      "logps/chosen": -109.0,
+      "logps/rejected": -62.5,
+      "loss": 0.8242,
+      "nll_loss": 1.2890625,
+      "rewards/accuracies": 0.75,
+      "rewards/chosen": -0.00390625,
+      "rewards/margins": 0.2255859375,
+      "rewards/rejected": -0.2294921875,
+      "step": 8
+    },
+    {
+      "epoch": 0.05263157894736842,
+      "grad_norm": 8.538987159729004,
+      "learning_rate": 1e-05,
+      "logits/chosen": 0.57421875,
+      "logits/rejected": 0.59375,
+      "logps/chosen": -69.0,
+      "logps/rejected": -5.875,
+      "loss": 0.8125,
+      "nll_loss": 1.484375,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 0.5,
+      "rewards/margins": 0.625,
+      "rewards/rejected": -0.125,
+      "step": 9
+    },
+    {
+      "epoch": 0.05847953216374269,
+      "grad_norm": 6.87076997756958,
+      "learning_rate": 9.999059852242508e-06,
+      "logits/chosen": 0.6484375,
+      "logits/rejected": 0.86328125,
+      "logps/chosen": -46.0,
+      "logps/rejected": -6.15625,
+      "loss": 0.7461,
+      "nll_loss": 1.390625,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 0.40625,
+      "rewards/margins": 0.5703125,
+      "rewards/rejected": -0.162109375,
+      "step": 10
+    },
+    {
+      "epoch": 0.06432748538011696,
+      "grad_norm": 6.21343469619751,
+      "learning_rate": 9.996239762521152e-06,
+      "logits/chosen": 0.12158203125,
+      "logits/rejected": 0.5234375,
+      "logps/chosen": -85.5,
+      "logps/rejected": -6.0,
+      "loss": 0.6478,
+      "nll_loss": 1.2421875,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 0.625,
+      "rewards/margins": 0.828125,
+      "rewards/rejected": -0.205078125,
+      "step": 11
+    },
+    {
+      "epoch": 0.07017543859649122,
+      "grad_norm": 5.582995891571045,
+      "learning_rate": 9.991540791356342e-06,
+      "logits/chosen": 0.208984375,
+      "logits/rejected": 0.0284423828125,
+      "logps/chosen": -76.5,
+      "logps/rejected": -20.0,
+      "loss": 0.5983,
+      "nll_loss": 1.4140625,
+      "rewards/accuracies": 0.75,
+      "rewards/chosen": 0.50390625,
+      "rewards/margins": 0.953125,
+      "rewards/rejected": -0.44921875,
+      "step": 12
+    },
+    {
+      "epoch": 0.07602339181286549,
+      "grad_norm": 4.830435752868652,
+      "learning_rate": 9.98496470583896e-06,
+      "logits/chosen": 0.318359375,
+      "logits/rejected": 0.68359375,
+      "logps/chosen": -70.5,
+      "logps/rejected": -10.75,
+      "loss": 0.5495,
+      "nll_loss": 1.2578125,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 1.09375,
+      "rewards/margins": 1.640625,
+      "rewards/rejected": -0.546875,
+      "step": 13
+    },
+    {
+      "epoch": 0.08187134502923976,
+      "grad_norm": 4.1727142333984375,
+      "learning_rate": 9.976513978965829e-06,
+      "logits/chosen": -0.029541015625,
+      "logits/rejected": 0.11474609375,
+      "logps/chosen": -135.0,
+      "logps/rejected": -17.375,
+      "loss": 0.4876,
+      "nll_loss": 1.3515625,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 1.6875,
+      "rewards/margins": 2.390625,
+      "rewards/rejected": -0.69921875,
+      "step": 14
+    },
+    {
+      "epoch": 0.08771929824561403,
+      "grad_norm": 2.942042827606201,
+      "learning_rate": 9.966191788709716e-06,
+      "logits/chosen": 0.298828125,
+      "logits/rejected": 0.3359375,
+      "logps/chosen": -70.0,
+      "logps/rejected": -15.1875,
+      "loss": 0.4977,
+      "nll_loss": 1.53125,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 0.875,
+      "rewards/margins": 1.640625,
+      "rewards/rejected": -0.76171875,
+      "step": 15
+    },
+    {
+      "epoch": 0.0935672514619883,
+      "grad_norm": 2.406817674636841,
+      "learning_rate": 9.954002016824226e-06,
+      "logits/chosen": 0.2890625,
+      "logits/rejected": 0.255859375,
+      "logps/chosen": -163.0,
+      "logps/rejected": -19.625,
+      "loss": 0.4163,
+      "nll_loss": 1.3125,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 2.75,
+      "rewards/margins": 3.671875,
+      "rewards/rejected": -0.91796875,
+      "step": 16
+    },
+    {
+      "epoch": 0.09941520467836257,
+      "grad_norm": 2.127260446548462,
+      "learning_rate": 9.939949247384046e-06,
+      "logits/chosen": 0.08984375,
+      "logits/rejected": 0.09765625,
+      "logps/chosen": -71.5,
+      "logps/rejected": -52.75,
+      "loss": 0.4284,
+      "nll_loss": 1.2734375,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 0.97265625,
+      "rewards/margins": 3.390625,
+      "rewards/rejected": -2.421875,
+      "step": 17
+    },
+    {
+      "epoch": 0.10526315789473684,
+      "grad_norm": 2.6332027912139893,
+      "learning_rate": 9.924038765061042e-06,
+      "logits/chosen": 0.19140625,
+      "logits/rejected": 0.04443359375,
+      "logps/chosen": -113.0,
+      "logps/rejected": -54.25,
+      "loss": 0.3786,
+      "nll_loss": 1.3046875,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 1.4375,
+      "rewards/margins": 3.9375,
+      "rewards/rejected": -2.515625,
+      "step": 18
+    },
+    {
+      "epoch": 0.1111111111111111,
+      "grad_norm": 2.0191080570220947,
+      "learning_rate": 9.906276553136924e-06,
+      "logits/chosen": 0.08544921875,
+      "logits/rejected": 0.49609375,
+      "logps/chosen": -88.0,
+      "logps/rejected": -8.3125,
+      "loss": 0.4479,
+      "nll_loss": 1.5546875,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 1.640625,
+      "rewards/margins": 1.8828125,
+      "rewards/rejected": -0.236328125,
+      "step": 19
+    },
+    {
+      "epoch": 0.11695906432748537,
+      "grad_norm": 1.9893779754638672,
+      "learning_rate": 9.886669291253178e-06,
+      "logits/chosen": 0.421875,
+      "logits/rejected": 0.349609375,
+      "logps/chosen": -34.25,
+      "logps/rejected": -7.625,
+      "loss": 0.4622,
+      "nll_loss": 1.1171875,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 0.57421875,
+      "rewards/margins": 1.7265625,
+      "rewards/rejected": -1.1484375,
+      "step": 20
+    },
+    {
+      "epoch": 0.12280701754385964,
+      "grad_norm": 1.3010271787643433,
+      "learning_rate": 9.86522435289912e-06,
+      "logits/chosen": 0.08740234375,
+      "logits/rejected": 0.359375,
+      "logps/chosen": -123.5,
+      "logps/rejected": -6.25,
+      "loss": 0.3633,
+      "nll_loss": 1.515625,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 3.265625,
+      "rewards/margins": 4.28125,
+      "rewards/rejected": -1.015625,
+      "step": 21
+    },
+    {
+      "epoch": 0.1286549707602339,
+      "grad_norm": 2.4463562965393066,
+      "learning_rate": 9.841949802639031e-06,
+      "logits/chosen": 0.2353515625,
+      "logits/rejected": 0.55859375,
+      "logps/chosen": -84.5,
+      "logps/rejected": -15.0,
+      "loss": 0.417,
+      "nll_loss": 1.125,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 1.4765625,
+      "rewards/margins": 3.09375,
+      "rewards/rejected": -1.625,
+      "step": 22
+    },
+    {
+      "epoch": 0.13450292397660818,
+      "grad_norm": 1.135468602180481,
+      "learning_rate": 9.816854393079402e-06,
+      "logits/chosen": 0.056396484375,
+      "logits/rejected": -0.01409912109375,
+      "logps/chosen": -67.0,
+      "logps/rejected": -22.25,
+      "loss": 0.3626,
+      "nll_loss": 1.4765625,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 0.443359375,
+      "rewards/margins": 2.46875,
+      "rewards/rejected": -2.03125,
+      "step": 23
+    },
+    {
+      "epoch": 0.14035087719298245,
+      "grad_norm": 1.4695147275924683,
+      "learning_rate": 9.789947561577445e-06,
+      "logits/chosen": -0.21484375,
+      "logits/rejected": 0.006256103515625,
+      "logps/chosen": -77.5,
+      "logps/rejected": -34.5,
+      "loss": 0.4385,
+      "nll_loss": 1.6171875,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 0.96875,
+      "rewards/margins": 2.828125,
+      "rewards/rejected": -1.859375,
+      "step": 24
+    },
+    {
+      "epoch": 0.14619883040935672,
+      "grad_norm": 0.9380862712860107,
+      "learning_rate": 9.761239426692077e-06,
+      "logits/chosen": 0.181640625,
+      "logits/rejected": 0.228515625,
+      "logps/chosen": -91.0,
+      "logps/rejected": -20.5,
+      "loss": 0.3815,
+      "nll_loss": 1.328125,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 2.390625,
+      "rewards/margins": 4.53125,
+      "rewards/rejected": -2.15625,
+      "step": 25
+    },
+    {
+      "epoch": 0.15204678362573099,
+      "grad_norm": 1.5922902822494507,
+      "learning_rate": 9.730740784378755e-06,
+      "logits/chosen": 0.40234375,
+      "logits/rejected": 0.46875,
+      "logps/chosen": -51.75,
+      "logps/rejected": -18.125,
+      "loss": 0.3636,
+      "nll_loss": 1.1640625,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 1.609375,
+      "rewards/margins": 2.84375,
+      "rewards/rejected": -1.234375,
+      "step": 26
+    },
+    {
+      "epoch": 0.15789473684210525,
+      "grad_norm": 1.7313437461853027,
+      "learning_rate": 9.698463103929542e-06,
+      "logits/chosen": 0.173828125,
+      "logits/rejected": 0.1943359375,
+      "logps/chosen": -94.0,
+      "logps/rejected": -38.25,
+      "loss": 0.3835,
+      "nll_loss": 1.265625,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 2.15625,
+      "rewards/margins": 5.84375,
+      "rewards/rejected": -3.6875,
+      "step": 27
+    },
+    {
+      "epoch": 0.16374269005847952,
+      "grad_norm": 1.2301464080810547,
+      "learning_rate": 9.664418523660004e-06,
+      "logits/chosen": 0.09228515625,
+      "logits/rejected": 0.1875,
+      "logps/chosen": -74.5,
+      "logps/rejected": -21.5,
+      "loss": 0.3408,
+      "nll_loss": 1.3203125,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 0.9765625,
+      "rewards/margins": 3.578125,
+      "rewards/rejected": -2.59375,
+      "step": 28
+    },
+    {
+      "epoch": 0.1695906432748538,
+      "grad_norm": 1.0106050968170166,
+      "learning_rate": 9.628619846344453e-06,
+      "logits/chosen": 0.62109375,
+      "logits/rejected": 0.5625,
+      "logps/chosen": -48.5,
+      "logps/rejected": -8.3125,
+      "loss": 0.3379,
+      "nll_loss": 1.5078125,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 2.6875,
+      "rewards/margins": 4.21875,
+      "rewards/rejected": -1.515625,
+      "step": 29
+    },
+    {
+      "epoch": 0.17543859649122806,
+      "grad_norm": 0.8290690183639526,
+      "learning_rate": 9.591080534401371e-06,
+      "logits/chosen": 0.1103515625,
+      "logits/rejected": 0.0712890625,
+      "logps/chosen": -103.0,
+      "logps/rejected": -28.125,
+      "loss": 0.3311,
+      "nll_loss": 1.4375,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 3.28125,
+      "rewards/margins": 6.5,
+      "rewards/rejected": -3.234375,
+      "step": 30
+    },
+    {
+      "epoch": 0.18128654970760233,
+      "grad_norm": 1.09013032913208,
+      "learning_rate": 9.551814704830734e-06,
+      "logits/chosen": 0.1494140625,
+      "logits/rejected": 0.90625,
+      "logps/chosen": -46.0,
+      "logps/rejected": -7.0,
+      "loss": 0.3809,
+      "nll_loss": 1.359375,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 1.9375,
+      "rewards/margins": 3.5625,
+      "rewards/rejected": -1.625,
+      "step": 31
+    },
+    {
+      "epoch": 0.1871345029239766,
+      "grad_norm": 0.8978441953659058,
+      "learning_rate": 9.51083712390519e-06,
+      "logits/chosen": 0.263671875,
+      "logits/rejected": 0.447265625,
+      "logps/chosen": -43.25,
+      "logps/rejected": -7.75,
+      "loss": 0.3184,
+      "nll_loss": 1.15625,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 0.8125,
+      "rewards/margins": 2.390625,
+      "rewards/rejected": -1.578125,
+      "step": 32
+    },
+    {
+      "epoch": 0.19298245614035087,
+      "grad_norm": 1.1078234910964966,
+      "learning_rate": 9.468163201617063e-06,
+      "logits/chosen": -0.08203125,
+      "logits/rejected": 0.25390625,
+      "logps/chosen": -120.0,
+      "logps/rejected": -19.875,
+      "loss": 0.3835,
+      "nll_loss": 1.71875,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 3.6875,
+      "rewards/margins": 6.75,
+      "rewards/rejected": -3.046875,
+      "step": 33
+    },
+    {
+      "epoch": 0.19883040935672514,
+      "grad_norm": 0.7379825711250305,
+      "learning_rate": 9.423808985883289e-06,
+      "logits/chosen": 0.19921875,
+      "logits/rejected": 0.46484375,
+      "logps/chosen": -96.0,
+      "logps/rejected": -30.5,
+      "loss": 0.383,
+      "nll_loss": 1.375,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 2.34375,
+      "rewards/margins": 5.3125,
+      "rewards/rejected": -2.953125,
+      "step": 34
+    },
+    {
+      "epoch": 0.2046783625730994,
+      "grad_norm": 1.2710590362548828,
+      "learning_rate": 9.377791156510456e-06,
+      "logits/chosen": -0.22265625,
+      "logits/rejected": -0.3984375,
+      "logps/chosen": -89.0,
+      "logps/rejected": -24.75,
+      "loss": 0.3789,
+      "nll_loss": 1.5859375,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 1.96875,
+      "rewards/margins": 5.21875,
+      "rewards/rejected": -3.25,
+      "step": 35
+    },
+    {
+      "epoch": 0.21052631578947367,
+      "grad_norm": 1.0742026567459106,
+      "learning_rate": 9.330127018922195e-06,
+      "logits/chosen": 0.0947265625,
+      "logits/rejected": 0.48828125,
+      "logps/chosen": -94.0,
+      "logps/rejected": -28.0,
+      "loss": 0.3975,
+      "nll_loss": 1.28125,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 0.9609375,
+      "rewards/margins": 4.1875,
+      "rewards/rejected": -3.21875,
+      "step": 36
+    },
+    {
+      "epoch": 0.21637426900584794,
+      "grad_norm": 0.5758035182952881,
+      "learning_rate": 9.280834497651334e-06,
+      "logits/chosen": -0.043212890625,
+      "logits/rejected": 0.1376953125,
+      "logps/chosen": -161.0,
+      "logps/rejected": -40.5,
+      "loss": 0.3301,
+      "nll_loss": 1.3515625,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 2.0625,
+      "rewards/margins": 6.9375,
+      "rewards/rejected": -4.875,
+      "step": 37
+    },
+    {
+      "epoch": 0.2222222222222222,
+      "grad_norm": 0.787466287612915,
+      "learning_rate": 9.229932129599206e-06,
+      "logits/chosen": 0.031494140625,
+      "logits/rejected": 0.4765625,
+      "logps/chosen": -58.0,
+      "logps/rejected": -9.4375,
+      "loss": 0.375,
+      "nll_loss": 1.4375,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 1.7265625,
+      "rewards/margins": 4.0,
+      "rewards/rejected": -2.28125,
+      "step": 38
+    },
+    {
+      "epoch": 0.22807017543859648,
+      "grad_norm": 0.9415507316589355,
+      "learning_rate": 9.177439057064684e-06,
+      "logits/chosen": 0.212890625,
+      "logits/rejected": 0.859375,
+      "logps/chosen": -37.5,
+      "logps/rejected": -4.21875,
+      "loss": 0.36,
+      "nll_loss": 1.171875,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 1.484375,
+      "rewards/margins": 2.828125,
+      "rewards/rejected": -1.3515625,
+      "step": 39
+    },
+    {
+      "epoch": 0.23391812865497075,
+      "grad_norm": 3.3659064769744873,
+      "learning_rate": 9.123375020545534e-06,
+      "logits/chosen": -0.0286865234375,
+      "logits/rejected": -0.0267333984375,
+      "logps/chosen": -67.0,
+      "logps/rejected": -33.5,
+      "loss": 0.3604,
+      "nll_loss": 1.1796875,
+      "rewards/accuracies": 0.75,
+      "rewards/chosen": -0.54296875,
+      "rewards/margins": 1.921875,
+      "rewards/rejected": -2.46875,
+      "step": 40
+    },
+    {
+      "epoch": 0.23976608187134502,
+      "grad_norm": 0.9403271675109863,
+      "learning_rate": 9.067760351314838e-06,
+      "logits/chosen": 0.1591796875,
+      "logits/rejected": 0.384765625,
+      "logps/chosen": -116.5,
+      "logps/rejected": -32.75,
+      "loss": 0.3714,
+      "nll_loss": 1.3125,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 0.15625,
+      "rewards/margins": 4.28125,
+      "rewards/rejected": -4.125,
+      "step": 41
+    },
+    {
+      "epoch": 0.24561403508771928,
+      "grad_norm": 0.7999655604362488,
+      "learning_rate": 9.01061596377522e-06,
+      "logits/chosen": -0.0654296875,
+      "logits/rejected": 0.37109375,
+      "logps/chosen": -102.0,
+      "logps/rejected": -11.1875,
+      "loss": 0.3747,
+      "nll_loss": 1.4921875,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 2.71875,
+      "rewards/margins": 5.125,
+      "rewards/rejected": -2.40625,
+      "step": 42
+    },
+    {
+      "epoch": 0.25146198830409355,
+      "grad_norm": 0.6711202263832092,
+      "learning_rate": 8.951963347593797e-06,
+      "logits/chosen": -0.016357421875,
+      "logits/rejected": -0.11767578125,
+      "logps/chosen": -85.5,
+      "logps/rejected": -28.5,
+      "loss": 0.3522,
+      "nll_loss": 1.21875,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 0.171875,
+      "rewards/margins": 4.71875,
+      "rewards/rejected": -4.5625,
+      "step": 43
+    },
+    {
+      "epoch": 0.2573099415204678,
+      "grad_norm": 0.550448477268219,
+      "learning_rate": 8.891824559620801e-06,
+      "logits/chosen": 0.03173828125,
+      "logits/rejected": -0.0693359375,
+      "logps/chosen": -116.0,
+      "logps/rejected": -52.75,
+      "loss": 0.3389,
+      "nll_loss": 1.4375,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 2.375,
+      "rewards/margins": 8.8125,
+      "rewards/rejected": -6.4375,
+      "step": 44
+    },
+    {
+      "epoch": 0.2631578947368421,
+      "grad_norm": 0.6417452096939087,
+      "learning_rate": 8.83022221559489e-06,
+      "logits/chosen": 0.3828125,
+      "logits/rejected": 0.376953125,
+      "logps/chosen": -68.5,
+      "logps/rejected": -11.8125,
+      "loss": 0.3363,
+      "nll_loss": 1.34375,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 2.25,
+      "rewards/margins": 4.5625,
+      "rewards/rejected": -2.3125,
+      "step": 45
+    },
+    {
+      "epoch": 0.26900584795321636,
+      "grad_norm": 1.0228935480117798,
+      "learning_rate": 8.767179481638303e-06,
+      "logits/chosen": 0.0311279296875,
+      "logits/rejected": 0.1357421875,
+      "logps/chosen": -119.0,
+      "logps/rejected": -57.75,
+      "loss": 0.3219,
+      "nll_loss": 1.171875,
+      "rewards/accuracies": 0.75,
+      "rewards/chosen": -0.875,
+      "rewards/margins": 5.375,
+      "rewards/rejected": -6.28125,
+      "step": 46
+    },
+    {
+      "epoch": 0.27485380116959063,
+      "grad_norm": 0.4747965335845947,
+      "learning_rate": 8.702720065545024e-06,
+      "logits/chosen": 0.2578125,
+      "logits/rejected": 0.3828125,
+      "logps/chosen": -73.0,
+      "logps/rejected": -31.375,
+      "loss": 0.2988,
+      "nll_loss": 1.1484375,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 2.421875,
+      "rewards/margins": 7.65625,
+      "rewards/rejected": -5.21875,
+      "step": 47
+    },
+    {
+      "epoch": 0.2807017543859649,
+      "grad_norm": 0.8870027661323547,
+      "learning_rate": 8.636868207865244e-06,
+      "logits/chosen": 0.177734375,
+      "logits/rejected": 0.201171875,
+      "logps/chosen": -97.0,
+      "logps/rejected": -14.5,
+      "loss": 0.3397,
+      "nll_loss": 1.3671875,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 2.5625,
+      "rewards/margins": 5.9375,
+      "rewards/rejected": -3.375,
+      "step": 48
+    },
+    {
+      "epoch": 0.28654970760233917,
+      "grad_norm": 0.7480011582374573,
+      "learning_rate": 8.569648672789496e-06,
+      "logits/chosen": -0.2060546875,
+      "logits/rejected": -0.11865234375,
+      "logps/chosen": -101.5,
+      "logps/rejected": -34.25,
+      "loss": 0.3369,
+      "nll_loss": 1.6328125,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 3.03125,
+      "rewards/margins": 7.0,
+      "rewards/rejected": -3.984375,
+      "step": 49
+    },
+    {
+      "epoch": 0.29239766081871343,
+      "grad_norm": 0.6633922457695007,
+      "learning_rate": 8.501086738835843e-06,
+      "logits/chosen": 0.19921875,
+      "logits/rejected": 0.353515625,
+      "logps/chosen": -115.5,
+      "logps/rejected": -47.25,
+      "loss": 0.3193,
+      "nll_loss": 1.421875,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 2.6875,
+      "rewards/margins": 8.9375,
+      "rewards/rejected": -6.25,
+      "step": 50
+    },
+    {
+      "epoch": 0.2982456140350877,
+      "grad_norm": 0.5760066509246826,
+      "learning_rate": 8.43120818934367e-06,
+      "logits/chosen": -0.06787109375,
+      "logits/rejected": 0.1962890625,
+      "logps/chosen": -126.0,
+      "logps/rejected": -22.0,
+      "loss": 0.3117,
+      "nll_loss": 1.59375,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 3.21875,
+      "rewards/margins": 7.125,
+      "rewards/rejected": -3.890625,
+      "step": 51
+    },
+    {
+      "epoch": 0.30409356725146197,
+      "grad_norm": 0.8805516958236694,
+      "learning_rate": 8.360039302777614e-06,
+      "logits/chosen": 0.03125,
+      "logits/rejected": 0.216796875,
+      "logps/chosen": -64.0,
+      "logps/rejected": -17.125,
+      "loss": 0.3623,
+      "nll_loss": 1.5546875,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 2.125,
+      "rewards/margins": 5.96875,
+      "rewards/rejected": -3.828125,
+      "step": 52
+    },
+    {
+      "epoch": 0.30994152046783624,
+      "grad_norm": 0.927228569984436,
+      "learning_rate": 8.28760684284532e-06,
+      "logits/chosen": 0.054931640625,
+      "logits/rejected": 0.16796875,
+      "logps/chosen": -94.0,
+      "logps/rejected": -58.0,
+      "loss": 0.3184,
+      "nll_loss": 1.1796875,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": -0.03515625,
+      "rewards/margins": 6.46875,
+      "rewards/rejected": -6.5,
+      "step": 53
+    },
+    {
+      "epoch": 0.3157894736842105,
+      "grad_norm": 1.1748754978179932,
+      "learning_rate": 8.213938048432697e-06,
+      "logits/chosen": 0.306640625,
+      "logits/rejected": 0.69140625,
+      "logps/chosen": -30.625,
+      "logps/rejected": -8.375,
+      "loss": 0.3766,
+      "nll_loss": 1.2265625,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 0.78125,
+      "rewards/margins": 3.1875,
+      "rewards/rejected": -2.40625,
+      "step": 54
+    },
+    {
+      "epoch": 0.3216374269005848,
+      "grad_norm": 1.0034699440002441,
+      "learning_rate": 8.139060623360494e-06,
+      "logits/chosen": 0.0478515625,
+      "logits/rejected": 0.1298828125,
+      "logps/chosen": -88.0,
+      "logps/rejected": -56.5,
+      "loss": 0.321,
+      "nll_loss": 1.2265625,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 1.0625,
+      "rewards/margins": 7.5,
+      "rewards/rejected": -6.4375,
+      "step": 55
+    },
+    {
+      "epoch": 0.32748538011695905,
+      "grad_norm": 1.2884156703948975,
+      "learning_rate": 8.063002725966014e-06,
+      "logits/chosen": -0.1767578125,
+      "logits/rejected": -0.1357421875,
+      "logps/chosen": -80.5,
+      "logps/rejected": -52.25,
+      "loss": 0.3215,
+      "nll_loss": 1.15625,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": -0.703125,
+      "rewards/margins": 5.71875,
+      "rewards/rejected": -6.40625,
+      "step": 56
+    },
+    {
+      "epoch": 0.3333333333333333,
+      "grad_norm": 0.9113926887512207,
+      "learning_rate": 7.985792958513932e-06,
+      "logits/chosen": 0.189453125,
+      "logits/rejected": 0.1650390625,
+      "logps/chosen": -59.0,
+      "logps/rejected": -26.75,
+      "loss": 0.3581,
+      "nll_loss": 1.2734375,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": -0.046875,
+      "rewards/margins": 3.8125,
+      "rewards/rejected": -3.875,
+      "step": 57
+    },
+    {
+      "epoch": 0.3391812865497076,
+      "grad_norm": 0.5659676194190979,
+      "learning_rate": 7.907460356440133e-06,
+      "logits/chosen": -0.042724609375,
+      "logits/rejected": 0.043212890625,
+      "logps/chosen": -91.5,
+      "logps/rejected": -41.5,
+      "loss": 0.3509,
+      "nll_loss": 1.6484375,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 2.75,
+      "rewards/margins": 8.9375,
+      "rewards/rejected": -6.21875,
+      "step": 58
+    },
+    {
+      "epoch": 0.34502923976608185,
+      "grad_norm": 0.5110981464385986,
+      "learning_rate": 7.828034377432694e-06,
+      "logits/chosen": 0.259765625,
+      "logits/rejected": 0.46875,
+      "logps/chosen": -23.25,
+      "logps/rejected": -12.625,
+      "loss": 0.2972,
+      "nll_loss": 1.0703125,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 0.0947265625,
+      "rewards/margins": 2.671875,
+      "rewards/rejected": -2.5625,
+      "step": 59
+    },
+    {
+      "epoch": 0.3508771929824561,
+      "grad_norm": 0.7066183686256409,
+      "learning_rate": 7.747544890354031e-06,
+      "logits/chosen": -0.00799560546875,
+      "logits/rejected": -0.049560546875,
+      "logps/chosen": -172.0,
+      "logps/rejected": -39.5,
+      "loss": 0.3223,
+      "nll_loss": 1.5078125,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 3.5,
+      "rewards/margins": 8.3125,
+      "rewards/rejected": -4.8125,
+      "step": 60
+    },
+    {
+      "epoch": 0.3567251461988304,
+      "grad_norm": 1.2593711614608765,
+      "learning_rate": 7.666022164008458e-06,
+      "logits/chosen": -0.123046875,
+      "logits/rejected": -0.00150299072265625,
+      "logps/chosen": -85.5,
+      "logps/rejected": -20.25,
+      "loss": 0.4053,
+      "nll_loss": 1.640625,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 1.828125,
+      "rewards/margins": 6.03125,
+      "rewards/rejected": -4.1875,
+      "step": 61
+    },
+    {
+      "epoch": 0.36257309941520466,
+      "grad_norm": 0.36089378595352173,
+      "learning_rate": 7.5834968557593155e-06,
+      "logits/chosen": 0.006591796875,
+      "logits/rejected": -0.1025390625,
+      "logps/chosen": -66.5,
+      "logps/rejected": -24.625,
+      "loss": 0.3109,
+      "nll_loss": 1.21875,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": -0.6640625,
+      "rewards/margins": 3.546875,
+      "rewards/rejected": -4.1875,
+      "step": 62
+    },
+    {
+      "epoch": 0.3684210526315789,
+      "grad_norm": 0.4141497015953064,
+      "learning_rate": 7.500000000000001e-06,
+      "logits/chosen": 0.0186767578125,
+      "logits/rejected": -0.00518798828125,
+      "logps/chosen": -93.5,
+      "logps/rejected": -44.75,
+      "loss": 0.2959,
+      "nll_loss": 1.3828125,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 0.03125,
+      "rewards/margins": 5.9375,
+      "rewards/rejected": -5.90625,
+      "step": 63
+    },
+    {
+      "epoch": 0.3742690058479532,
+      "grad_norm": 0.21456825733184814,
+      "learning_rate": 7.415562996483193e-06,
+      "logits/chosen": 0.205078125,
+      "logits/rejected": 0.019775390625,
+      "logps/chosen": -58.25,
+      "logps/rejected": -21.5,
+      "loss": 0.2822,
+      "nll_loss": 1.2265625,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 3.125,
+      "rewards/margins": 6.9375,
+      "rewards/rejected": -3.828125,
+      "step": 64
+    },
+    {
+      "epoch": 0.38011695906432746,
+      "grad_norm": 0.5265213251113892,
+      "learning_rate": 7.330217598512696e-06,
+      "logits/chosen": 0.177734375,
+      "logits/rejected": 0.447265625,
+      "logps/chosen": -78.0,
+      "logps/rejected": -14.0,
+      "loss": 0.3379,
+      "nll_loss": 1.3671875,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 2.03125,
+      "rewards/margins": 5.4375,
+      "rewards/rejected": -3.390625,
+      "step": 65
+    },
+    {
+      "epoch": 0.38596491228070173,
+      "grad_norm": 0.46037861704826355,
+      "learning_rate": 7.243995901002312e-06,
+      "logits/chosen": 0.142578125,
+      "logits/rejected": 0.041748046875,
+      "logps/chosen": -107.0,
+      "logps/rejected": -39.0,
+      "loss": 0.3047,
+      "nll_loss": 1.421875,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 1.25,
+      "rewards/margins": 7.28125,
+      "rewards/rejected": -6.03125,
+      "step": 66
+    },
+    {
+      "epoch": 0.391812865497076,
+      "grad_norm": 0.7725083231925964,
+      "learning_rate": 7.156930328406268e-06,
+      "logits/chosen": -0.130859375,
+      "logits/rejected": -0.0228271484375,
+      "logps/chosen": -126.0,
+      "logps/rejected": -62.0,
+      "loss": 0.3267,
+      "nll_loss": 1.28125,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 0.4375,
+      "rewards/margins": 8.125,
+      "rewards/rejected": -7.6875,
+      "step": 67
+    },
+    {
+      "epoch": 0.39766081871345027,
+      "grad_norm": 0.8484850525856018,
+      "learning_rate": 7.069053622525697e-06,
+      "logits/chosen": -0.020263671875,
+      "logits/rejected": 0.369140625,
+      "logps/chosen": -95.5,
+      "logps/rejected": -31.5,
+      "loss": 0.3545,
+      "nll_loss": 1.5859375,
+      "rewards/accuracies": 0.75,
+      "rewards/chosen": 1.75,
+      "rewards/margins": 5.4375,
+      "rewards/rejected": -3.6875,
+      "step": 68
+    },
+    {
+      "epoch": 0.40350877192982454,
+      "grad_norm": 0.7983413338661194,
+      "learning_rate": 6.980398830195785e-06,
+      "logits/chosen": 0.296875,
+      "logits/rejected": 0.255859375,
+      "logps/chosen": -34.5,
+      "logps/rejected": -22.875,
+      "loss": 0.3704,
+      "nll_loss": 1.2265625,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 0.33203125,
+      "rewards/margins": 3.53125,
+      "rewards/rejected": -3.1875,
+      "step": 69
+    },
+    {
+      "epoch": 0.4093567251461988,
+      "grad_norm": 0.6347444653511047,
+      "learning_rate": 6.890999290858213e-06,
+      "logits/chosen": 0.1904296875,
+      "logits/rejected": 0.330078125,
+      "logps/chosen": -78.5,
+      "logps/rejected": -36.25,
+      "loss": 0.3239,
+      "nll_loss": 1.1640625,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 0.0703125,
+      "rewards/margins": 5.78125,
+      "rewards/rejected": -5.6875,
+      "step": 70
+    },
+    {
+      "epoch": 0.4152046783625731,
+      "grad_norm": 1.4213659763336182,
+      "learning_rate": 6.800888624023552e-06,
+      "logits/chosen": -0.048583984375,
+      "logits/rejected": -0.310546875,
+      "logps/chosen": -72.0,
+      "logps/rejected": -39.5,
+      "loss": 0.3571,
+      "nll_loss": 1.4609375,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 2.25,
+      "rewards/margins": 6.75,
+      "rewards/rejected": -4.5,
+      "step": 71
+    },
+    {
+      "epoch": 0.42105263157894735,
+      "grad_norm": 0.8988905549049377,
+      "learning_rate": 6.710100716628345e-06,
+      "logits/chosen": 0.1845703125,
+      "logits/rejected": -0.0242919921875,
+      "logps/chosen": -59.0,
+      "logps/rejected": -21.25,
+      "loss": 0.3424,
+      "nll_loss": 1.3125,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": -0.0625,
+      "rewards/margins": 2.78125,
+      "rewards/rejected": -2.84375,
+      "step": 72
+    },
+    {
+      "epoch": 0.4269005847953216,
+      "grad_norm": 0.47313231229782104,
+      "learning_rate": 6.618669710291607e-06,
+      "logits/chosen": 0.11279296875,
+      "logits/rejected": 0.10986328125,
+      "logps/chosen": -105.5,
+      "logps/rejected": -47.25,
+      "loss": 0.3132,
+      "nll_loss": 1.515625,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 3.75,
+      "rewards/margins": 10.0,
+      "rewards/rejected": -6.21875,
+      "step": 73
+    },
+    {
+      "epoch": 0.4327485380116959,
+      "grad_norm": 0.5204113721847534,
+      "learning_rate": 6.526629988475567e-06,
+      "logits/chosen": 0.10400390625,
+      "logits/rejected": 0.000659942626953125,
+      "logps/chosen": -99.0,
+      "logps/rejected": -30.125,
+      "loss": 0.2962,
+      "nll_loss": 1.359375,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 3.125,
+      "rewards/margins": 7.59375,
+      "rewards/rejected": -4.46875,
+      "step": 74
+    },
+    {
+      "epoch": 0.43859649122807015,
+      "grad_norm": 0.5024166703224182,
+      "learning_rate": 6.434016163555452e-06,
+      "logits/chosen": 0.140625,
+      "logits/rejected": 0.51171875,
+      "logps/chosen": -94.5,
+      "logps/rejected": -30.625,
+      "loss": 0.2926,
+      "nll_loss": 1.34375,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 1.984375,
+      "rewards/margins": 5.90625,
+      "rewards/rejected": -3.90625,
+      "step": 75
+    },
+    {
+      "epoch": 0.4444444444444444,
+      "grad_norm": 0.8214257955551147,
+      "learning_rate": 6.340863063803187e-06,
+      "logits/chosen": 0.1650390625,
+      "logits/rejected": 0.4296875,
+      "logps/chosen": -69.0,
+      "logps/rejected": -20.25,
+      "loss": 0.3695,
+      "nll_loss": 1.546875,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 1.2890625,
+      "rewards/margins": 4.0625,
+      "rewards/rejected": -2.78125,
+      "step": 76
+    },
+    {
+      "epoch": 0.4502923976608187,
+      "grad_norm": 0.5306276082992554,
+      "learning_rate": 6.247205720289907e-06,
+      "logits/chosen": 0.2353515625,
+      "logits/rejected": 0.28515625,
+      "logps/chosen": -90.5,
+      "logps/rejected": -10.6875,
+      "loss": 0.3353,
+      "nll_loss": 1.5234375,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 3.09375,
+      "rewards/margins": 6.46875,
+      "rewards/rejected": -3.375,
+      "step": 77
+    },
+    {
+      "epoch": 0.45614035087719296,
+      "grad_norm": 0.7573875188827515,
+      "learning_rate": 6.153079353712201e-06,
+      "logits/chosen": -0.24609375,
+      "logits/rejected": -0.33203125,
+      "logps/chosen": -75.0,
+      "logps/rejected": -40.5,
+      "loss": 0.3079,
+      "nll_loss": 1.3046875,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 0.2734375,
+      "rewards/margins": 5.125,
+      "rewards/rejected": -4.84375,
+      "step": 78
+    },
+    {
+      "epoch": 0.4619883040935672,
+      "grad_norm": 0.8065016865730286,
+      "learning_rate": 6.058519361147055e-06,
+      "logits/chosen": 0.240234375,
+      "logits/rejected": 0.318359375,
+      "logps/chosen": -79.5,
+      "logps/rejected": -13.5,
+      "loss": 0.3405,
+      "nll_loss": 1.453125,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 3.625,
+      "rewards/margins": 7.03125,
+      "rewards/rejected": -3.40625,
+      "step": 79
+    },
+    {
+      "epoch": 0.4678362573099415,
+      "grad_norm": 0.41012996435165405,
+      "learning_rate": 5.9635613027404495e-06,
+      "logits/chosen": 0.09326171875,
+      "logits/rejected": 0.02490234375,
+      "logps/chosen": -95.0,
+      "logps/rejected": -31.75,
+      "loss": 0.3265,
+      "nll_loss": 1.5546875,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 3.9375,
+      "rewards/margins": 8.4375,
+      "rewards/rejected": -4.5,
+      "step": 80
+    },
+    {
+      "epoch": 0.47368421052631576,
+      "grad_norm": 0.5796492695808411,
+      "learning_rate": 5.8682408883346535e-06,
+      "logits/chosen": 0.2490234375,
+      "logits/rejected": 0.59765625,
+      "logps/chosen": -61.25,
+      "logps/rejected": -9.0,
+      "loss": 0.3291,
+      "nll_loss": 1.375,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 1.8125,
+      "rewards/margins": 4.125,
+      "rewards/rejected": -2.296875,
+      "step": 81
+    },
+    {
+      "epoch": 0.47953216374269003,
+      "grad_norm": 0.37575745582580566,
+      "learning_rate": 5.772593964039203e-06,
+      "logits/chosen": -0.2119140625,
+      "logits/rejected": -0.00066375732421875,
+      "logps/chosen": -164.0,
+      "logps/rejected": -44.0,
+      "loss": 0.2995,
+      "nll_loss": 1.4453125,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 4.3125,
+      "rewards/margins": 10.8125,
+      "rewards/rejected": -6.5,
+      "step": 82
+    },
+    {
+      "epoch": 0.4853801169590643,
+      "grad_norm": 0.37623926997184753,
+      "learning_rate": 5.6766564987506564e-06,
+      "logits/chosen": 0.173828125,
+      "logits/rejected": -0.07568359375,
+      "logps/chosen": -68.5,
+      "logps/rejected": -42.0,
+      "loss": 0.3045,
+      "nll_loss": 1.1796875,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 0.171875,
+      "rewards/margins": 6.0625,
+      "rewards/rejected": -5.875,
+      "step": 83
+    },
+    {
+      "epoch": 0.49122807017543857,
+      "grad_norm": 0.6429110765457153,
+      "learning_rate": 5.5804645706261515e-06,
+      "logits/chosen": -0.18359375,
+      "logits/rejected": 0.1103515625,
+      "logps/chosen": -124.5,
+      "logps/rejected": -25.75,
+      "loss": 0.3444,
+      "nll_loss": 1.46875,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 2.6875,
+      "rewards/margins": 7.125,
+      "rewards/rejected": -4.4375,
+      "step": 84
+    },
+    {
+      "epoch": 0.49707602339181284,
+      "grad_norm": 0.5742049217224121,
+      "learning_rate": 5.484054353515896e-06,
+      "logits/chosen": -0.150390625,
+      "logits/rejected": 0.2001953125,
+      "logps/chosen": -97.0,
+      "logps/rejected": -52.25,
+      "loss": 0.3678,
+      "nll_loss": 1.328125,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 0.4375,
+      "rewards/margins": 8.8125,
+      "rewards/rejected": -8.375,
+      "step": 85
+    },
+    {
+      "epoch": 0.5029239766081871,
+      "grad_norm": 0.42503976821899414,
+      "learning_rate": 5.387462103359655e-06,
+      "logits/chosen": 0.2216796875,
+      "logits/rejected": 0.34375,
+      "logps/chosen": -91.0,
+      "logps/rejected": -15.5625,
+      "loss": 0.2835,
+      "nll_loss": 1.3515625,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 5.40625,
+      "rewards/margins": 9.625,
+      "rewards/rejected": -4.21875,
+      "step": 86
+    },
+    {
+      "epoch": 0.5087719298245614,
+      "grad_norm": 1.4887723922729492,
+      "learning_rate": 5.290724144552379e-06,
+      "logits/chosen": -0.34375,
+      "logits/rejected": -0.177734375,
+      "logps/chosen": -106.0,
+      "logps/rejected": -34.0,
+      "loss": 0.3737,
+      "nll_loss": 1.4765625,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 1.375,
+      "rewards/margins": 6.125,
+      "rewards/rejected": -4.75,
+      "step": 87
+    },
+    {
+      "epoch": 0.5146198830409356,
+      "grad_norm": 0.6670490503311157,
+      "learning_rate": 5.193876856284085e-06,
+      "logits/chosen": 0.310546875,
+      "logits/rejected": 0.3359375,
+      "logps/chosen": -34.0,
+      "logps/rejected": -19.5,
+      "loss": 0.3307,
+      "nll_loss": 1.03125,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 1.703125,
+      "rewards/margins": 5.75,
+      "rewards/rejected": -4.0625,
+      "step": 88
+    },
+    {
+      "epoch": 0.52046783625731,
+      "grad_norm": 0.2873896062374115,
+      "learning_rate": 5.096956658859122e-06,
+      "logits/chosen": -0.08544921875,
+      "logits/rejected": -0.06494140625,
+      "logps/chosen": -79.5,
+      "logps/rejected": -17.25,
+      "loss": 0.2904,
+      "nll_loss": 1.3671875,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 1.953125,
+      "rewards/margins": 5.90625,
+      "rewards/rejected": -3.953125,
+      "step": 89
+    },
+    {
+      "epoch": 0.5263157894736842,
+      "grad_norm": 0.8809191584587097,
+      "learning_rate": 5e-06,
+      "logits/chosen": -0.07861328125,
+      "logits/rejected": -0.076171875,
+      "logps/chosen": -183.0,
+      "logps/rejected": -44.0,
+      "loss": 0.3906,
+      "nll_loss": 1.484375,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 2.0,
+      "rewards/margins": 7.90625,
+      "rewards/rejected": -5.90625,
+      "step": 90
+    },
+    {
+      "epoch": 0.5321637426900585,
+      "grad_norm": 0.6138753294944763,
+      "learning_rate": 4.903043341140879e-06,
+      "logits/chosen": -0.03369140625,
+      "logits/rejected": 0.054443359375,
+      "logps/chosen": -115.0,
+      "logps/rejected": -17.375,
+      "loss": 0.3411,
+      "nll_loss": 1.53125,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 1.53125,
+      "rewards/margins": 5.3125,
+      "rewards/rejected": -3.78125,
+      "step": 91
+    },
+    {
+      "epoch": 0.5380116959064327,
+      "grad_norm": 0.5989710092544556,
+      "learning_rate": 4.806123143715916e-06,
+      "logits/chosen": 0.068359375,
+      "logits/rejected": -0.091796875,
+      "logps/chosen": -109.5,
+      "logps/rejected": -29.0,
+      "loss": 0.3449,
+      "nll_loss": 1.3671875,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 1.28125,
+      "rewards/margins": 6.21875,
+      "rewards/rejected": -4.9375,
+      "step": 92
+    },
+    {
+      "epoch": 0.543859649122807,
+      "grad_norm": 0.7193765640258789,
+      "learning_rate": 4.7092758554476215e-06,
+      "logits/chosen": 0.40625,
+      "logits/rejected": 0.25,
+      "logps/chosen": -60.75,
+      "logps/rejected": -18.875,
+      "loss": 0.3236,
+      "nll_loss": 1.4453125,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 1.53125,
+      "rewards/margins": 5.59375,
+      "rewards/rejected": -4.0625,
+      "step": 93
+    },
+    {
+      "epoch": 0.5497076023391813,
+      "grad_norm": 0.86937016248703,
+      "learning_rate": 4.6125378966403465e-06,
+      "logits/chosen": 0.12890625,
+      "logits/rejected": 0.142578125,
+      "logps/chosen": -70.5,
+      "logps/rejected": -19.75,
+      "loss": 0.373,
+      "nll_loss": 1.2421875,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 1.5,
+      "rewards/margins": 5.875,
+      "rewards/rejected": -4.375,
+      "step": 94
+    },
+    {
+      "epoch": 0.5555555555555556,
+      "grad_norm": 0.5262789130210876,
+      "learning_rate": 4.515945646484105e-06,
+      "logits/chosen": -0.040771484375,
+      "logits/rejected": -0.048095703125,
+      "logps/chosen": -67.0,
+      "logps/rejected": -27.0,
+      "loss": 0.3229,
+      "nll_loss": 1.2109375,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": -0.046875,
+      "rewards/margins": 4.84375,
+      "rewards/rejected": -4.875,
+      "step": 95
+    },
+    {
+      "epoch": 0.5614035087719298,
+      "grad_norm": 0.3275640904903412,
+      "learning_rate": 4.4195354293738484e-06,
+      "logits/chosen": -0.279296875,
+      "logits/rejected": -0.072265625,
+      "logps/chosen": -186.0,
+      "logps/rejected": -62.25,
+      "loss": 0.2866,
+      "nll_loss": 1.4453125,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 2.375,
+      "rewards/margins": 9.3125,
+      "rewards/rejected": -6.9375,
+      "step": 96
+    },
+    {
+      "epoch": 0.5672514619883041,
+      "grad_norm": 0.7497672438621521,
+      "learning_rate": 4.323343501249346e-06,
+      "logits/chosen": 0.20703125,
+      "logits/rejected": 0.296875,
+      "logps/chosen": -123.5,
+      "logps/rejected": -23.5,
+      "loss": 0.349,
+      "nll_loss": 1.625,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 4.25,
+      "rewards/margins": 8.875,
+      "rewards/rejected": -4.625,
+      "step": 97
+    },
+    {
+      "epoch": 0.5730994152046783,
+      "grad_norm": 0.7181825637817383,
+      "learning_rate": 4.227406035960798e-06,
+      "logits/chosen": -0.18359375,
+      "logits/rejected": -0.2177734375,
+      "logps/chosen": -92.5,
+      "logps/rejected": -29.25,
+      "loss": 0.335,
+      "nll_loss": 1.5625,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 2.515625,
+      "rewards/margins": 7.46875,
+      "rewards/rejected": -4.96875,
+      "step": 98
+    },
+    {
+      "epoch": 0.5789473684210527,
+      "grad_norm": 0.4383862614631653,
+      "learning_rate": 4.131759111665349e-06,
+      "logits/chosen": 0.0218505859375,
+      "logits/rejected": 0.373046875,
+      "logps/chosen": -61.25,
+      "logps/rejected": -10.0625,
+      "loss": 0.3148,
+      "nll_loss": 1.5859375,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 2.484375,
+      "rewards/margins": 5.0625,
+      "rewards/rejected": -2.578125,
+      "step": 99
+    },
+    {
+      "epoch": 0.5847953216374269,
+      "grad_norm": 0.29338470101356506,
+      "learning_rate": 4.036438697259551e-06,
+      "logits/chosen": 0.283203125,
+      "logits/rejected": 0.32421875,
+      "logps/chosen": -100.0,
+      "logps/rejected": -23.375,
+      "loss": 0.2728,
+      "nll_loss": 1.3046875,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 1.75,
+      "rewards/margins": 6.53125,
+      "rewards/rejected": -4.78125,
+      "step": 100
+    },
+    {
+      "epoch": 0.5906432748538012,
+      "grad_norm": 1.3003735542297363,
+      "learning_rate": 3.941480638852948e-06,
+      "logits/chosen": -0.1484375,
+      "logits/rejected": -0.0830078125,
+      "logps/chosen": -106.0,
+      "logps/rejected": -31.875,
+      "loss": 0.3271,
+      "nll_loss": 1.40625,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 0.78125,
+      "rewards/margins": 6.5625,
+      "rewards/rejected": -5.78125,
+      "step": 101
+    },
+    {
+      "epoch": 0.5964912280701754,
+      "grad_norm": 0.961757481098175,
+      "learning_rate": 3.8469206462878e-06,
+      "logits/chosen": -0.0947265625,
+      "logits/rejected": 0.11376953125,
+      "logps/chosen": -119.5,
+      "logps/rejected": -43.5,
+      "loss": 0.347,
+      "nll_loss": 1.484375,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 2.640625,
+      "rewards/margins": 8.625,
+      "rewards/rejected": -6.0,
+      "step": 102
+    },
+    {
+      "epoch": 0.6023391812865497,
+      "grad_norm": 1.0032845735549927,
+      "learning_rate": 3.752794279710094e-06,
+      "logits/chosen": 0.205078125,
+      "logits/rejected": 0.255859375,
+      "logps/chosen": -59.0,
+      "logps/rejected": -33.0,
+      "loss": 0.3779,
+      "nll_loss": 1.328125,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 2.140625,
+      "rewards/margins": 6.5625,
+      "rewards/rejected": -4.4375,
+      "step": 103
+    },
+    {
+      "epoch": 0.6081871345029239,
+      "grad_norm": 0.7629097104072571,
+      "learning_rate": 3.6591369361968127e-06,
+      "logits/chosen": 0.0030975341796875,
+      "logits/rejected": 0.1552734375,
+      "logps/chosen": -113.0,
+      "logps/rejected": -31.875,
+      "loss": 0.3118,
+      "nll_loss": 1.5234375,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 3.8125,
+      "rewards/margins": 7.53125,
+      "rewards/rejected": -3.703125,
+      "step": 104
+    },
+    {
+      "epoch": 0.6140350877192983,
+      "grad_norm": 0.7119747996330261,
+      "learning_rate": 3.5659838364445505e-06,
+      "logits/chosen": 0.050537109375,
+      "logits/rejected": -0.09228515625,
+      "logps/chosen": -74.5,
+      "logps/rejected": -36.5,
+      "loss": 0.2962,
+      "nll_loss": 1.3203125,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 2.875,
+      "rewards/margins": 8.875,
+      "rewards/rejected": -5.96875,
+      "step": 105
+    },
+    {
+      "epoch": 0.6198830409356725,
+      "grad_norm": 0.7210653424263,
+      "learning_rate": 3.473370011524435e-06,
+      "logits/chosen": 0.5234375,
+      "logits/rejected": 0.3515625,
+      "logps/chosen": -35.75,
+      "logps/rejected": -14.625,
+      "loss": 0.3092,
+      "nll_loss": 1.140625,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 0.74609375,
+      "rewards/margins": 3.65625,
+      "rewards/rejected": -2.90625,
+      "step": 106
+    },
+    {
+      "epoch": 0.6257309941520468,
+      "grad_norm": 0.6818527579307556,
+      "learning_rate": 3.3813302897083955e-06,
+      "logits/chosen": 0.1337890625,
+      "logits/rejected": 0.19921875,
+      "logps/chosen": -122.5,
+      "logps/rejected": -45.5,
+      "loss": 0.3034,
+      "nll_loss": 1.328125,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 2.5,
+      "rewards/margins": 7.84375,
+      "rewards/rejected": -5.34375,
+      "step": 107
+    },
+    {
+      "epoch": 0.631578947368421,
+      "grad_norm": 0.4697604179382324,
+      "learning_rate": 3.289899283371657e-06,
+      "logits/chosen": -0.2451171875,
+      "logits/rejected": 0.1630859375,
+      "logps/chosen": -112.5,
+      "logps/rejected": -34.25,
+      "loss": 0.3125,
+      "nll_loss": 1.359375,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 0.5,
+      "rewards/margins": 6.53125,
+      "rewards/rejected": -6.03125,
+      "step": 108
+    },
+    {
+      "epoch": 0.6374269005847953,
+      "grad_norm": 0.971885085105896,
+      "learning_rate": 3.1991113759764493e-06,
+      "logits/chosen": 0.0439453125,
+      "logits/rejected": 0.5546875,
+      "logps/chosen": -85.0,
+      "logps/rejected": -11.875,
+      "loss": 0.3936,
+      "nll_loss": 1.4375,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 1.96875,
+      "rewards/margins": 5.5,
+      "rewards/rejected": -3.515625,
+      "step": 109
+    },
+    {
+      "epoch": 0.6432748538011696,
+      "grad_norm": 0.36553868651390076,
+      "learning_rate": 3.1090007091417884e-06,
+      "logits/chosen": 0.07080078125,
+      "logits/rejected": 0.04736328125,
+      "logps/chosen": -94.0,
+      "logps/rejected": -55.25,
+      "loss": 0.2773,
+      "nll_loss": 1.328125,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 1.96875,
+      "rewards/margins": 8.5625,
+      "rewards/rejected": -6.59375,
+      "step": 110
+    },
+    {
+      "epoch": 0.6491228070175439,
+      "grad_norm": 0.8389025926589966,
+      "learning_rate": 3.019601169804216e-06,
+      "logits/chosen": 0.1552734375,
+      "logits/rejected": 0.373046875,
+      "logps/chosen": -118.5,
+      "logps/rejected": -34.75,
+      "loss": 0.3086,
+      "nll_loss": 1.2578125,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 2.40625,
+      "rewards/margins": 7.40625,
+      "rewards/rejected": -5.0,
+      "step": 111
+    },
+    {
+      "epoch": 0.6549707602339181,
+      "grad_norm": 0.8012142777442932,
+      "learning_rate": 2.9309463774743047e-06,
+      "logits/chosen": 0.06298828125,
+      "logits/rejected": 0.1875,
+      "logps/chosen": -92.5,
+      "logps/rejected": -52.25,
+      "loss": 0.3294,
+      "nll_loss": 1.40625,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": -0.390625,
+      "rewards/margins": 6.25,
+      "rewards/rejected": -6.625,
+      "step": 112
+    },
+    {
+      "epoch": 0.6608187134502924,
+      "grad_norm": 0.6787911057472229,
+      "learning_rate": 2.843069671593734e-06,
+      "logits/chosen": 0.2333984375,
+      "logits/rejected": 0.201171875,
+      "logps/chosen": -92.0,
+      "logps/rejected": -17.5,
+      "loss": 0.3617,
+      "nll_loss": 1.4140625,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 3.078125,
+      "rewards/margins": 7.125,
+      "rewards/rejected": -4.03125,
+      "step": 113
+    },
+    {
+      "epoch": 0.6666666666666666,
+      "grad_norm": 0.7218255996704102,
+      "learning_rate": 2.7560040989976894e-06,
+      "logits/chosen": -0.031982421875,
+      "logits/rejected": -0.03662109375,
+      "logps/chosen": -70.0,
+      "logps/rejected": -36.75,
+      "loss": 0.3638,
+      "nll_loss": 1.125,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 1.71875,
+      "rewards/margins": 5.59375,
+      "rewards/rejected": -3.859375,
+      "step": 114
+    },
+    {
+      "epoch": 0.672514619883041,
+      "grad_norm": 0.6574802994728088,
+      "learning_rate": 2.6697824014873076e-06,
+      "logits/chosen": 0.1806640625,
+      "logits/rejected": 0.02001953125,
+      "logps/chosen": -101.0,
+      "logps/rejected": -31.75,
+      "loss": 0.318,
+      "nll_loss": 1.3359375,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 0.65625,
+      "rewards/margins": 6.625,
+      "rewards/rejected": -5.9375,
+      "step": 115
+    },
+    {
+      "epoch": 0.6783625730994152,
+      "grad_norm": 0.8116902709007263,
+      "learning_rate": 2.5844370035168077e-06,
+      "logits/chosen": 0.1162109375,
+      "logits/rejected": 0.2470703125,
+      "logps/chosen": -88.5,
+      "logps/rejected": -63.0,
+      "loss": 0.306,
+      "nll_loss": 1.2109375,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": -1.71875,
+      "rewards/margins": 5.59375,
+      "rewards/rejected": -7.3125,
+      "step": 116
+    },
+    {
+      "epoch": 0.6842105263157895,
+      "grad_norm": 0.6500887870788574,
+      "learning_rate": 2.5000000000000015e-06,
+      "logits/chosen": -0.046875,
+      "logits/rejected": 0.1630859375,
+      "logps/chosen": -96.0,
+      "logps/rejected": -35.25,
+      "loss": 0.29,
+      "nll_loss": 1.234375,
+      "rewards/accuracies": 0.75,
+      "rewards/chosen": -0.875,
+      "rewards/margins": 3.90625,
+      "rewards/rejected": -4.78125,
+      "step": 117
+    },
+    {
+      "epoch": 0.6900584795321637,
+      "grad_norm": 1.0178558826446533,
+      "learning_rate": 2.4165031442406857e-06,
+      "logits/chosen": 0.041748046875,
+      "logits/rejected": 0.2421875,
+      "logps/chosen": -53.0,
+      "logps/rejected": -15.0625,
+      "loss": 0.3434,
+      "nll_loss": 1.484375,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 2.640625,
+      "rewards/margins": 6.21875,
+      "rewards/rejected": -3.5625,
+      "step": 118
+    },
+    {
+      "epoch": 0.695906432748538,
+      "grad_norm": 0.7249758243560791,
+      "learning_rate": 2.333977835991545e-06,
+      "logits/chosen": 0.28125,
+      "logits/rejected": 0.087890625,
+      "logps/chosen": -81.5,
+      "logps/rejected": -32.25,
+      "loss": 0.2837,
+      "nll_loss": 1.140625,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 2.375,
+      "rewards/margins": 5.875,
+      "rewards/rejected": -3.46875,
+      "step": 119
+    },
+    {
+      "epoch": 0.7017543859649122,
+      "grad_norm": 0.6112239360809326,
+      "learning_rate": 2.2524551096459703e-06,
+      "logits/chosen": 0.18359375,
+      "logits/rejected": 0.2236328125,
+      "logps/chosen": -78.0,
+      "logps/rejected": -30.0,
+      "loss": 0.3184,
+      "nll_loss": 1.40625,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 1.34375,
+      "rewards/margins": 5.71875,
+      "rewards/rejected": -4.375,
+      "step": 120
+    },
+    {
+      "epoch": 0.7076023391812866,
+      "grad_norm": 0.9316398501396179,
+      "learning_rate": 2.171965622567308e-06,
+      "logits/chosen": 0.396484375,
+      "logits/rejected": 0.1953125,
+      "logps/chosen": -72.5,
+      "logps/rejected": -41.0,
+      "loss": 0.3656,
+      "nll_loss": 1.1640625,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": -0.90234375,
+      "rewards/margins": 3.625,
+      "rewards/rejected": -4.53125,
+      "step": 121
+    },
+    {
+      "epoch": 0.7134502923976608,
+      "grad_norm": 2.40132474899292,
+      "learning_rate": 2.0925396435598665e-06,
+      "logits/chosen": -0.01092529296875,
+      "logits/rejected": 0.376953125,
+      "logps/chosen": -50.75,
+      "logps/rejected": -15.0,
+      "loss": 0.3434,
+      "nll_loss": 1.515625,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 2.140625,
+      "rewards/margins": 5.46875,
+      "rewards/rejected": -3.3125,
+      "step": 122
+    },
+    {
+      "epoch": 0.7192982456140351,
+      "grad_norm": 0.5527243614196777,
+      "learning_rate": 2.0142070414860704e-06,
+      "logits/chosen": 0.1416015625,
+      "logits/rejected": 0.25,
+      "logps/chosen": -117.5,
+      "logps/rejected": -78.0,
+      "loss": 0.3078,
+      "nll_loss": 1.2421875,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": -0.3125,
+      "rewards/margins": 8.375,
+      "rewards/rejected": -8.75,
+      "step": 123
+    },
+    {
+      "epoch": 0.7251461988304093,
+      "grad_norm": 0.6802273392677307,
+      "learning_rate": 1.936997274033986e-06,
+      "logits/chosen": 0.271484375,
+      "logits/rejected": 0.2431640625,
+      "logps/chosen": -110.5,
+      "logps/rejected": -35.75,
+      "loss": 0.2995,
+      "nll_loss": 1.3671875,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 3.375,
+      "rewards/margins": 9.0,
+      "rewards/rejected": -5.625,
+      "step": 124
+    },
+    {
+      "epoch": 0.7309941520467836,
+      "grad_norm": 0.270517498254776,
+      "learning_rate": 1.8609393766395083e-06,
+      "logits/chosen": 0.2431640625,
+      "logits/rejected": 0.020751953125,
+      "logps/chosen": -109.0,
+      "logps/rejected": -29.25,
+      "loss": 0.277,
+      "nll_loss": 1.3828125,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 2.59375,
+      "rewards/margins": 8.25,
+      "rewards/rejected": -5.6875,
+      "step": 125
+    },
+    {
+      "epoch": 0.7368421052631579,
+      "grad_norm": 0.4582943320274353,
+      "learning_rate": 1.7860619515673034e-06,
+      "logits/chosen": 0.107421875,
+      "logits/rejected": 0.197265625,
+      "logps/chosen": -113.0,
+      "logps/rejected": -49.25,
+      "loss": 0.2612,
+      "nll_loss": 1.4453125,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 0.75,
+      "rewards/margins": 6.84375,
+      "rewards/rejected": -6.09375,
+      "step": 126
+    },
+    {
+      "epoch": 0.7426900584795322,
+      "grad_norm": 1.0266200304031372,
+      "learning_rate": 1.7123931571546826e-06,
+      "logits/chosen": 0.158203125,
+      "logits/rejected": 0.006988525390625,
+      "logps/chosen": -126.0,
+      "logps/rejected": -32.25,
+      "loss": 0.3486,
+      "nll_loss": 1.375,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 4.125,
+      "rewards/margins": 9.9375,
+      "rewards/rejected": -5.84375,
+      "step": 127
+    },
+    {
+      "epoch": 0.7485380116959064,
+      "grad_norm": 0.3515218198299408,
+      "learning_rate": 1.639960697222388e-06,
+      "logits/chosen": -0.357421875,
+      "logits/rejected": -0.02783203125,
+      "logps/chosen": -98.0,
+      "logps/rejected": -31.0,
+      "loss": 0.2879,
+      "nll_loss": 1.640625,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 1.59375,
+      "rewards/margins": 6.65625,
+      "rewards/rejected": -5.0625,
+      "step": 128
+    },
+    {
+      "epoch": 0.7543859649122807,
+      "grad_norm": 0.9205822944641113,
+      "learning_rate": 1.5687918106563326e-06,
+      "logits/chosen": -0.123046875,
+      "logits/rejected": 0.251953125,
+      "logps/chosen": -116.5,
+      "logps/rejected": -12.0,
+      "loss": 0.3135,
+      "nll_loss": 1.6796875,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 4.53125,
+      "rewards/margins": 7.125,
+      "rewards/rejected": -2.609375,
+      "step": 129
+    },
+    {
+      "epoch": 0.7602339181286549,
+      "grad_norm": 0.47208088636398315,
+      "learning_rate": 1.4989132611641576e-06,
+      "logits/chosen": -0.10791015625,
+      "logits/rejected": -0.1298828125,
+      "logps/chosen": -90.0,
+      "logps/rejected": -28.75,
+      "loss": 0.349,
+      "nll_loss": 1.375,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 4.03125,
+      "rewards/margins": 9.25,
+      "rewards/rejected": -5.1875,
+      "step": 130
+    },
+    {
+      "epoch": 0.7660818713450293,
+      "grad_norm": 1.2003756761550903,
+      "learning_rate": 1.4303513272105057e-06,
+      "logits/chosen": -0.1357421875,
+      "logits/rejected": -0.1318359375,
+      "logps/chosen": -171.0,
+      "logps/rejected": -63.75,
+      "loss": 0.3398,
+      "nll_loss": 1.5390625,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 4.34375,
+      "rewards/margins": 13.1875,
+      "rewards/rejected": -8.875,
+      "step": 131
+    },
+    {
+      "epoch": 0.7719298245614035,
+      "grad_norm": 0.6289705634117126,
+      "learning_rate": 1.3631317921347564e-06,
+      "logits/chosen": -0.181640625,
+      "logits/rejected": 0.203125,
+      "logps/chosen": -75.0,
+      "logps/rejected": -23.75,
+      "loss": 0.3607,
+      "nll_loss": 1.1171875,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": -0.671875,
+      "rewards/margins": 4.46875,
+      "rewards/rejected": -5.125,
+      "step": 132
+    },
+    {
+      "epoch": 0.7777777777777778,
+      "grad_norm": 0.5018782615661621,
+      "learning_rate": 1.297279934454978e-06,
+      "logits/chosen": -0.0537109375,
+      "logits/rejected": -0.0615234375,
+      "logps/chosen": -100.5,
+      "logps/rejected": -31.125,
+      "loss": 0.3402,
+      "nll_loss": 1.1875,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 0.71875,
+      "rewards/margins": 5.34375,
+      "rewards/rejected": -4.625,
+      "step": 133
+    },
+    {
+      "epoch": 0.783625730994152,
+      "grad_norm": 0.7222105860710144,
+      "learning_rate": 1.2328205183616964e-06,
+      "logits/chosen": 0.1474609375,
+      "logits/rejected": 0.0311279296875,
+      "logps/chosen": -96.5,
+      "logps/rejected": -29.5,
+      "loss": 0.3115,
+      "nll_loss": 1.359375,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 1.4765625,
+      "rewards/margins": 6.53125,
+      "rewards/rejected": -5.0625,
+      "step": 134
+    },
+    {
+      "epoch": 0.7894736842105263,
+      "grad_norm": 0.9635083079338074,
+      "learning_rate": 1.1697777844051105e-06,
+      "logits/chosen": 0.51953125,
+      "logits/rejected": 0.427734375,
+      "logps/chosen": -75.0,
+      "logps/rejected": -20.0,
+      "loss": 0.3672,
+      "nll_loss": 1.3203125,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 1.046875,
+      "rewards/margins": 4.71875,
+      "rewards/rejected": -3.65625,
+      "step": 135
+    },
+    {
+      "epoch": 0.7953216374269005,
+      "grad_norm": 1.1779509782791138,
+      "learning_rate": 1.1081754403792e-06,
+      "logits/chosen": 0.0859375,
+      "logits/rejected": 0.59765625,
+      "logps/chosen": -61.0,
+      "logps/rejected": -10.5,
+      "loss": 0.3646,
+      "nll_loss": 1.578125,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 1.046875,
+      "rewards/margins": 3.34375,
+      "rewards/rejected": -2.296875,
+      "step": 136
+    },
+    {
+      "epoch": 0.8011695906432749,
+      "grad_norm": 0.40560975670814514,
+      "learning_rate": 1.0480366524062041e-06,
+      "logits/chosen": -0.037353515625,
+      "logits/rejected": 0.02392578125,
+      "logps/chosen": -88.5,
+      "logps/rejected": -35.5,
+      "loss": 0.3294,
+      "nll_loss": 1.3359375,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 0.9375,
+      "rewards/margins": 6.28125,
+      "rewards/rejected": -5.375,
+      "step": 137
+    },
+    {
+      "epoch": 0.8070175438596491,
+      "grad_norm": 1.7967106103897095,
+      "learning_rate": 9.893840362247809e-07,
+      "logits/chosen": 0.1484375,
+      "logits/rejected": 0.21484375,
+      "logps/chosen": -82.0,
+      "logps/rejected": -40.75,
+      "loss": 0.3294,
+      "nll_loss": 1.421875,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 3.6875,
+      "rewards/margins": 7.71875,
+      "rewards/rejected": -4.03125,
+      "step": 138
+    },
+    {
+      "epoch": 0.8128654970760234,
+      "grad_norm": 0.5179945230484009,
+      "learning_rate": 9.322396486851626e-07,
+      "logits/chosen": -0.20703125,
+      "logits/rejected": 0.333984375,
+      "logps/chosen": -122.0,
+      "logps/rejected": -31.0,
+      "loss": 0.3402,
+      "nll_loss": 1.3359375,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 1.8125,
+      "rewards/margins": 6.625,
+      "rewards/rejected": -4.8125,
+      "step": 139
+    },
+    {
+      "epoch": 0.8187134502923976,
+      "grad_norm": 0.6503077149391174,
+      "learning_rate": 8.766249794544662e-07,
+      "logits/chosen": -0.140625,
+      "logits/rejected": 0.4609375,
+      "logps/chosen": -80.5,
+      "logps/rejected": -10.625,
+      "loss": 0.3496,
+      "nll_loss": 1.453125,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 1.734375,
+      "rewards/margins": 4.6875,
+      "rewards/rejected": -2.96875,
+      "step": 140
+    },
+    {
+      "epoch": 0.8245614035087719,
+      "grad_norm": 0.37708234786987305,
+      "learning_rate": 8.225609429353187e-07,
+      "logits/chosen": 0.09326171875,
+      "logits/rejected": 0.08203125,
+      "logps/chosen": -103.0,
+      "logps/rejected": -14.75,
+      "loss": 0.3428,
+      "nll_loss": 1.515625,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 2.1875,
+      "rewards/margins": 6.15625,
+      "rewards/rejected": -3.96875,
+      "step": 141
+    },
+    {
+      "epoch": 0.8304093567251462,
+      "grad_norm": 0.580976128578186,
+      "learning_rate": 7.700678704007947e-07,
+      "logits/chosen": -0.005645751953125,
+      "logits/rejected": 0.3125,
+      "logps/chosen": -83.0,
+      "logps/rejected": -21.75,
+      "loss": 0.3499,
+      "nll_loss": 1.4453125,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 3.578125,
+      "rewards/margins": 7.90625,
+      "rewards/rejected": -4.3125,
+      "step": 142
+    },
+    {
+      "epoch": 0.8362573099415205,
+      "grad_norm": 0.6109355092048645,
+      "learning_rate": 7.191655023486682e-07,
+      "logits/chosen": -0.19921875,
+      "logits/rejected": -0.064453125,
+      "logps/chosen": -84.0,
+      "logps/rejected": -24.625,
+      "loss": 0.3216,
+      "nll_loss": 1.515625,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 3.375,
+      "rewards/margins": 8.3125,
+      "rewards/rejected": -4.9375,
+      "step": 143
+    },
+    {
+      "epoch": 0.8421052631578947,
+      "grad_norm": 1.3205121755599976,
+      "learning_rate": 6.698729810778065e-07,
+      "logits/chosen": 0.349609375,
+      "logits/rejected": 0.107421875,
+      "logps/chosen": -60.0,
+      "logps/rejected": -31.0,
+      "loss": 0.3433,
+      "nll_loss": 1.1640625,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 0.43359375,
+      "rewards/margins": 5.09375,
+      "rewards/rejected": -4.65625,
+      "step": 144
+    },
+    {
+      "epoch": 0.847953216374269,
+      "grad_norm": 0.23541516065597534,
+      "learning_rate": 6.222088434895462e-07,
+      "logits/chosen": 0.251953125,
+      "logits/rejected": 0.2333984375,
+      "logps/chosen": -82.0,
+      "logps/rejected": -41.0,
+      "loss": 0.2943,
+      "nll_loss": 1.1328125,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 0.78125,
+      "rewards/margins": 6.84375,
+      "rewards/rejected": -6.0625,
+      "step": 145
+    },
+    {
+      "epoch": 0.8538011695906432,
+      "grad_norm": 1.0106936693191528,
+      "learning_rate": 5.76191014116711e-07,
+      "logits/chosen": -0.119140625,
+      "logits/rejected": 0.314453125,
+      "logps/chosen": -62.5,
+      "logps/rejected": -20.875,
+      "loss": 0.3275,
+      "nll_loss": 1.2109375,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 0.8125,
+      "rewards/margins": 4.53125,
+      "rewards/rejected": -3.71875,
+      "step": 146
+    },
+    {
+      "epoch": 0.8596491228070176,
+      "grad_norm": 0.31476834416389465,
+      "learning_rate": 5.318367983829393e-07,
+      "logits/chosen": 0.0291748046875,
+      "logits/rejected": -0.0272216796875,
+      "logps/chosen": -132.0,
+      "logps/rejected": -49.25,
+      "loss": 0.3047,
+      "nll_loss": 1.3828125,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 1.71875,
+      "rewards/margins": 9.0,
+      "rewards/rejected": -7.28125,
+      "step": 147
+    },
+    {
+      "epoch": 0.8654970760233918,
+      "grad_norm": 0.5657747387886047,
+      "learning_rate": 4.891628760948114e-07,
+      "logits/chosen": 0.06640625,
+      "logits/rejected": -0.05712890625,
+      "logps/chosen": -68.0,
+      "logps/rejected": -12.875,
+      "loss": 0.3312,
+      "nll_loss": 1.53125,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 3.375,
+      "rewards/margins": 6.53125,
+      "rewards/rejected": -3.15625,
+      "step": 148
+    },
+    {
+      "epoch": 0.8713450292397661,
+      "grad_norm": 0.4523642063140869,
+      "learning_rate": 4.481852951692672e-07,
+      "logits/chosen": -0.1015625,
+      "logits/rejected": 0.125,
+      "logps/chosen": -91.0,
+      "logps/rejected": -35.75,
+      "loss": 0.3071,
+      "nll_loss": 1.328125,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 1.625,
+      "rewards/margins": 6.6875,
+      "rewards/rejected": -5.0625,
+      "step": 149
+    },
+    {
+      "epoch": 0.8771929824561403,
+      "grad_norm": 0.5197721123695374,
+      "learning_rate": 4.089194655986306e-07,
+      "logits/chosen": 0.40234375,
+      "logits/rejected": 0.2197265625,
+      "logps/chosen": -41.25,
+      "logps/rejected": -17.5,
+      "loss": 0.3177,
+      "nll_loss": 1.0703125,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 0.296875,
+      "rewards/margins": 4.71875,
+      "rewards/rejected": -4.40625,
+      "step": 150
+    },
+    {
+      "epoch": 0.8830409356725146,
+      "grad_norm": 0.5756105184555054,
+      "learning_rate": 3.7138015365554834e-07,
+      "logits/chosen": -0.1953125,
+      "logits/rejected": -0.05615234375,
+      "logps/chosen": -93.0,
+      "logps/rejected": -32.0,
+      "loss": 0.3623,
+      "nll_loss": 1.609375,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 2.5,
+      "rewards/margins": 7.25,
+      "rewards/rejected": -4.75,
+      "step": 151
+    },
+    {
+      "epoch": 0.8888888888888888,
+      "grad_norm": 0.8583580255508423,
+      "learning_rate": 3.355814763399973e-07,
+      "logits/chosen": 0.388671875,
+      "logits/rejected": 0.67578125,
+      "logps/chosen": -24.875,
+      "logps/rejected": -10.125,
+      "loss": 0.3177,
+      "nll_loss": 0.97265625,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 0.30078125,
+      "rewards/margins": 3.28125,
+      "rewards/rejected": -2.96875,
+      "step": 152
+    },
+    {
+      "epoch": 0.8947368421052632,
+      "grad_norm": 0.9073471426963806,
+      "learning_rate": 3.015368960704584e-07,
+      "logits/chosen": 0.2041015625,
+      "logits/rejected": 0.14453125,
+      "logps/chosen": -88.0,
+      "logps/rejected": -33.25,
+      "loss": 0.2979,
+      "nll_loss": 1.421875,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 4.5,
+      "rewards/margins": 8.875,
+      "rewards/rejected": -4.375,
+      "step": 153
+    },
+    {
+      "epoch": 0.9005847953216374,
+      "grad_norm": 0.7385669350624084,
+      "learning_rate": 2.6925921562124867e-07,
+      "logits/chosen": -0.2021484375,
+      "logits/rejected": -0.017333984375,
+      "logps/chosen": -136.0,
+      "logps/rejected": -47.25,
+      "loss": 0.3239,
+      "nll_loss": 1.4375,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 1.15625,
+      "rewards/margins": 6.75,
+      "rewards/rejected": -5.59375,
+      "step": 154
+    },
+    {
+      "epoch": 0.9064327485380117,
+      "grad_norm": 0.2535851299762726,
+      "learning_rate": 2.3876057330792344e-07,
+      "logits/chosen": 0.134765625,
+      "logits/rejected": 0.578125,
+      "logps/chosen": -69.5,
+      "logps/rejected": -12.875,
+      "loss": 0.3099,
+      "nll_loss": 1.6484375,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 3.8125,
+      "rewards/margins": 7.09375,
+      "rewards/rejected": -3.28125,
+      "step": 155
+    },
+    {
+      "epoch": 0.9122807017543859,
+      "grad_norm": 0.9155167937278748,
+      "learning_rate": 2.1005243842255552e-07,
+      "logits/chosen": 0.341796875,
+      "logits/rejected": 0.045654296875,
+      "logps/chosen": -39.75,
+      "logps/rejected": -22.25,
+      "loss": 0.3564,
+      "nll_loss": 1.3828125,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 1.3828125,
+      "rewards/margins": 5.15625,
+      "rewards/rejected": -3.765625,
+      "step": 156
+    },
+    {
+      "epoch": 0.9181286549707602,
+      "grad_norm": 0.6450337171554565,
+      "learning_rate": 1.8314560692059836e-07,
+      "logits/chosen": 0.05419921875,
+      "logits/rejected": 0.10302734375,
+      "logps/chosen": -67.5,
+      "logps/rejected": -11.3125,
+      "loss": 0.319,
+      "nll_loss": 1.4765625,
+      "rewards/accuracies": 0.75,
+      "rewards/chosen": 2.28125,
+      "rewards/margins": 5.25,
+      "rewards/rejected": -2.96875,
+      "step": 157
+    },
+    {
+      "epoch": 0.9239766081871345,
+      "grad_norm": 0.455644816160202,
+      "learning_rate": 1.5805019736097105e-07,
+      "logits/chosen": 0.11572265625,
+      "logits/rejected": 0.126953125,
+      "logps/chosen": -125.5,
+      "logps/rejected": -31.25,
+      "loss": 0.3109,
+      "nll_loss": 1.375,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 0.5625,
+      "rewards/margins": 6.21875,
+      "rewards/rejected": -5.65625,
+      "step": 158
+    },
+    {
+      "epoch": 0.9298245614035088,
+      "grad_norm": 0.5387133359909058,
+      "learning_rate": 1.3477564710088097e-07,
+      "logits/chosen": 0.0093994140625,
+      "logits/rejected": -0.1943359375,
+      "logps/chosen": -100.0,
+      "logps/rejected": -55.75,
+      "loss": 0.3451,
+      "nll_loss": 1.421875,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 0.84375,
+      "rewards/margins": 8.3125,
+      "rewards/rejected": -7.4375,
+      "step": 159
+    },
+    {
+      "epoch": 0.935672514619883,
+      "grad_norm": 0.8238602876663208,
+      "learning_rate": 1.1333070874682217e-07,
+      "logits/chosen": 0.3046875,
+      "logits/rejected": -0.048583984375,
+      "logps/chosen": -28.25,
+      "logps/rejected": -23.125,
+      "loss": 0.3132,
+      "nll_loss": 1.109375,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 0.609375,
+      "rewards/margins": 5.0625,
+      "rewards/rejected": -4.46875,
+      "step": 160
+    },
+    {
+      "epoch": 0.9415204678362573,
+      "grad_norm": 0.33214011788368225,
+      "learning_rate": 9.372344686307655e-08,
+      "logits/chosen": 0.11865234375,
+      "logits/rejected": 0.2177734375,
+      "logps/chosen": -75.0,
+      "logps/rejected": -25.25,
+      "loss": 0.3014,
+      "nll_loss": 1.3359375,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 4.03125,
+      "rewards/margins": 8.3125,
+      "rewards/rejected": -4.28125,
+      "step": 161
+    },
+    {
+      "epoch": 0.9473684210526315,
+      "grad_norm": 0.7948828935623169,
+      "learning_rate": 7.59612349389599e-08,
+      "logits/chosen": 0.1044921875,
+      "logits/rejected": 0.029296875,
+      "logps/chosen": -131.0,
+      "logps/rejected": -52.5,
+      "loss": 0.3363,
+      "nll_loss": 1.484375,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": -0.1875,
+      "rewards/margins": 6.4375,
+      "rewards/rejected": -6.625,
+      "step": 162
+    },
+    {
+      "epoch": 0.9532163742690059,
+      "grad_norm": 0.8551905155181885,
+      "learning_rate": 6.005075261595495e-08,
+      "logits/chosen": 0.2265625,
+      "logits/rejected": 0.1142578125,
+      "logps/chosen": -86.0,
+      "logps/rejected": -78.0,
+      "loss": 0.3298,
+      "nll_loss": 1.28125,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": -0.296875,
+      "rewards/margins": 6.625,
+      "rewards/rejected": -6.90625,
+      "step": 163
+    },
+    {
+      "epoch": 0.9590643274853801,
+      "grad_norm": 0.8062785267829895,
+      "learning_rate": 4.599798317577342e-08,
+      "logits/chosen": 0.341796875,
+      "logits/rejected": 0.12890625,
+      "logps/chosen": -49.75,
+      "logps/rejected": -17.25,
+      "loss": 0.3372,
+      "nll_loss": 1.34375,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 2.84375,
+      "rewards/margins": 7.84375,
+      "rewards/rejected": -5.0,
+      "step": 164
+    },
+    {
+      "epoch": 0.9649122807017544,
+      "grad_norm": 0.8692733645439148,
+      "learning_rate": 3.3808211290284886e-08,
+      "logits/chosen": 0.40234375,
+      "logits/rejected": 0.435546875,
+      "logps/chosen": -47.75,
+      "logps/rejected": -16.375,
+      "loss": 0.36,
+      "nll_loss": 1.4453125,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 2.796875,
+      "rewards/margins": 6.09375,
+      "rewards/rejected": -3.3125,
+      "step": 165
+    },
+    {
+      "epoch": 0.9707602339181286,
+      "grad_norm": 1.0079779624938965,
+      "learning_rate": 2.3486021034170857e-08,
+      "logits/chosen": 0.020263671875,
+      "logits/rejected": 0.1279296875,
+      "logps/chosen": -95.5,
+      "logps/rejected": -28.25,
+      "loss": 0.3105,
+      "nll_loss": 1.5625,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 3.546875,
+      "rewards/margins": 8.4375,
+      "rewards/rejected": -4.90625,
+      "step": 166
+    },
+    {
+      "epoch": 0.9766081871345029,
+      "grad_norm": 0.6318630576133728,
+      "learning_rate": 1.5035294161039882e-08,
+      "logits/chosen": 0.279296875,
+      "logits/rejected": 0.1279296875,
+      "logps/chosen": -65.5,
+      "logps/rejected": -33.0,
+      "loss": 0.3509,
+      "nll_loss": 1.3203125,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 0.40625,
+      "rewards/margins": 5.96875,
+      "rewards/rejected": -5.5625,
+      "step": 167
+    },
+    {
+      "epoch": 0.9824561403508771,
+      "grad_norm": 0.29909488558769226,
+      "learning_rate": 8.459208643659122e-09,
+      "logits/chosen": 0.125,
+      "logits/rejected": -0.1298828125,
+      "logps/chosen": -95.5,
+      "logps/rejected": -25.25,
+      "loss": 0.3252,
+      "nll_loss": 1.234375,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 2.375,
+      "rewards/margins": 6.3125,
+      "rewards/rejected": -3.9375,
+      "step": 168
+    },
+    {
+      "epoch": 0.9883040935672515,
+      "grad_norm": 0.5387401580810547,
+      "learning_rate": 3.760237478849793e-09,
+      "logits/chosen": 0.212890625,
+      "logits/rejected": 0.26171875,
+      "logps/chosen": -50.5,
+      "logps/rejected": -7.9375,
+      "loss": 0.3005,
+      "nll_loss": 1.25,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 2.734375,
+      "rewards/margins": 5.6875,
+      "rewards/rejected": -2.953125,
+      "step": 169
+    },
+    {
+      "epoch": 0.9941520467836257,
+      "grad_norm": 0.8090826272964478,
+      "learning_rate": 9.401477574932927e-10,
+      "logits/chosen": 0.1474609375,
+      "logits/rejected": 0.0162353515625,
+      "logps/chosen": -85.5,
+      "logps/rejected": -35.75,
+      "loss": 0.2936,
+      "nll_loss": 1.2734375,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 1.25,
+      "rewards/margins": 6.40625,
+      "rewards/rejected": -5.15625,
+      "step": 170
+    },
+    {
+      "epoch": 1.0,
+      "grad_norm": 0.5961074829101562,
+      "learning_rate": 0.0,
+      "logits/chosen": 0.41796875,
+      "logits/rejected": 0.220703125,
+      "logps/chosen": -35.0,
+      "logps/rejected": -24.0,
+      "loss": 0.2952,
+      "nll_loss": 0.89453125,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": -0.75390625,
+      "rewards/margins": 4.15625,
+      "rewards/rejected": -4.90625,
+      "step": 171
+    }
+  ],
+  "logging_steps": 1,
+  "max_steps": 171,
+  "num_input_tokens_seen": 0,
+  "num_train_epochs": 1,
+  "save_steps": 500,
+  "stateful_callbacks": {
+    "TrainerControl": {
+      "args": {
+        "should_epoch_stop": false,
+        "should_evaluate": false,
+        "should_log": false,
+        "should_save": true,
+        "should_training_stop": true
+      },
+      "attributes": {}
+    }
+  },
+  "total_flos": 0.0,
+  "train_batch_size": 4,
+  "trial_name": null,
+  "trial_params": null
+}

7b/iteration1/training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0c4877ccc02fb1ffdfff8ba648f988fe56ad16f63dd5cc8d2f636b43152bfbba
+size 8056

7b/iteration2/README.md

@@ -0,0 +1,202 @@
+---
+base_model: pretrain_model/llava-1.5-7b-hf
+library_name: peft
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.14.0

7b/iteration2/adapter_config.json

@@ -0,0 +1,130 @@
+{
+  "alpha_pattern": {},
+  "auto_mapping": null,
+  "base_model_name_or_path": "pretrain_model/llava-1.5-7b-hf",
+  "bias": "none",
+  "eva_config": null,
+  "exclude_modules": null,
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 256,
+  "lora_bias": false,
+  "lora_dropout": 0.05,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "r": 128,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "24.self_attn.q_proj",
+    "model.layers.13.self_attn.v_proj",
+    "model.layers.3.self_attn.k_proj",
+    "model.layers.14.self_attn.k_proj",
+    "model.layers.13.self_attn.k_proj",
+    "model.layers.20.self_attn.q_proj",
+    "model.layers.16.self_attn.k_proj",
+    "28.self_attn.q_proj",
+    "model.layers.14.self_attn.q_proj",
+    "25.self_attn.k_proj",
+    "model.layers.7.self_attn.q_proj",
+    "model.layers.21.self_attn.k_proj",
+    "model.layers.14.self_attn.v_proj",
+    "26.self_attn.q_proj",
+    "30.self_attn.q_proj",
+    "model.layers.23.self_attn.k_proj",
+    "25.self_attn.q_proj",
+    "model.layers.8.self_attn.k_proj",
+    "model.layers.9.self_attn.v_proj",
+    "model.layers.9.self_attn.q_proj",
+    "model.layers.17.self_attn.q_proj",
+    "26.self_attn.k_proj",
+    "25.self_attn.v_proj",
+    "24.self_attn.v_proj",
+    "model.layers.13.self_attn.q_proj",
+    "model.layers.23.self_attn.v_proj",
+    "30.self_attn.k_proj",
+    "model.layers.1.self_attn.v_proj",
+    "model.layers.9.self_attn.k_proj",
+    "model.layers.5.self_attn.k_proj",
+    "model.layers.11.self_attn.q_proj",
+    "31.self_attn.k_proj",
+    "model.layers.15.self_attn.v_proj",
+    "model.layers.22.self_attn.q_proj",
+    "model.layers.1.self_attn.k_proj",
+    "model.layers.17.self_attn.v_proj",
+    "model.layers.12.self_attn.k_proj",
+    "model.layers.0.self_attn.q_proj",
+    "model.layers.19.self_attn.q_proj",
+    "model.layers.18.self_attn.k_proj",
+    "model.layers.23.self_attn.q_proj",
+    "27.self_attn.k_proj",
+    "up_proj",
+    "29.self_attn.v_proj",
+    "27.self_attn.q_proj",
+    "31.self_attn.v_proj",
+    "model.layers.12.self_attn.q_proj",
+    "model.layers.22.self_attn.v_proj",
+    "gate_proj",
+    "model.layers.8.self_attn.q_proj",
+    "model.layers.6.self_attn.v_proj",
+    "model.layers.16.self_attn.q_proj",
+    "model.layers.4.self_attn.q_proj",
+    "model.layers.19.self_attn.v_proj",
+    "model.layers.8.self_attn.v_proj",
+    "model.layers.18.self_attn.v_proj",
+    "model.layers.15.self_attn.q_proj",
+    "model.layers.1.self_attn.q_proj",
+    "model.layers.19.self_attn.k_proj",
+    "model.layers.2.self_attn.v_proj",
+    "o_proj",
+    "model.layers.10.self_attn.v_proj",
+    "model.layers.4.self_attn.k_proj",
+    "model.layers.5.self_attn.v_proj",
+    "model.layers.15.self_attn.k_proj",
+    "down_proj",
+    "24.self_attn.k_proj",
+    "model.layers.2.self_attn.k_proj",
+    "model.layers.21.self_attn.v_proj",
+    "model.layers.6.self_attn.q_proj",
+    "model.layers.3.self_attn.q_proj",
+    "27.self_attn.v_proj",
+    "model.layers.11.self_attn.k_proj",
+    "30.self_attn.v_proj",
+    "model.layers.7.self_attn.k_proj",
+    "model.layers.10.self_attn.k_proj",
+    "model.layers.10.self_attn.q_proj",
+    "29.self_attn.q_proj",
+    "31.self_attn.q_proj",
+    "28.self_attn.k_proj",
+    "26.self_attn.v_proj",
+    "model.layers.20.self_attn.k_proj",
+    "model.layers.2.self_attn.q_proj",
+    "model.layers.21.self_attn.q_proj",
+    "model.layers.12.self_attn.v_proj",
+    "28.self_attn.v_proj",
+    "model.layers.17.self_attn.k_proj",
+    "29.self_attn.k_proj",
+    "model.layers.5.self_attn.q_proj",
+    "model.layers.20.self_attn.v_proj",
+    "model.layers.7.self_attn.v_proj",
+    "model.layers.0.self_attn.v_proj",
+    "model.layers.6.self_attn.k_proj",
+    "model.layers.16.self_attn.v_proj",
+    "model.layers.11.self_attn.v_proj",
+    "model.layers.0.self_attn.k_proj",
+    "model.layers.22.self_attn.k_proj",
+    "model.layers.4.self_attn.v_proj",
+    "model.layers.18.self_attn.q_proj",
+    "model.layers.3.self_attn.v_proj"
+  ],
+  "task_type": "CAUSAL_LM",
+  "use_dora": false,
+  "use_rslora": false
+}

7b/iteration2/adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6af4787eaf6a4528a69021075ed5bf7253a581cdf53d1f7beee79575a6dd59d1
+size 639699488

7b/iteration2/added_tokens.json

@@ -0,0 +1,4 @@
+{
+  "<image>": 32000,
+  "<pad>": 32001
+}

7b/iteration2/chat_template.json

@@ -0,0 +1,3 @@
+{
+  "chat_template": "{% for message in messages %}{% if message['role'] != 'system' %}{{ message['role'].upper() + ': '}}{% endif %}{# Render all images first #}{% for content in message['content'] | selectattr('type', 'equalto', 'image') %}{{ '<image>\n' }}{% endfor %}{# Render all text next #}{% if message['role'] != 'assistant' %}{% for content in message['content'] | selectattr('type', 'equalto', 'text') %}{{ content['text'] + ' '}}{% endfor %}{% else %}{% for content in message['content'] | selectattr('type', 'equalto', 'text') %}{% generation %}{{ content['text'] + ' '}}{% endgeneration %}{% endfor %}{% endif %}{% endfor %}{% if add_generation_prompt %}{{ 'ASSISTANT:' }}{% endif %}"
+}

7b/iteration2/preprocessor_config.json

@@ -0,0 +1,28 @@
+{
+  "crop_size": {
+    "height": 336,
+    "width": 336
+  },
+  "do_center_crop": true,
+  "do_convert_rgb": true,
+  "do_normalize": true,
+  "do_rescale": true,
+  "do_resize": true,
+  "image_mean": [
+    0.48145466,
+    0.4578275,
+    0.40821073
+  ],
+  "image_processor_type": "CLIPImageProcessor",
+  "image_std": [
+    0.26862954,
+    0.26130258,
+    0.27577711
+  ],
+  "processor_class": "LlavaProcessor",
+  "resample": 3,
+  "rescale_factor": 0.00392156862745098,
+  "size": {
+    "shortest_edge": 336
+  }
+}

7b/iteration2/processor_config.json

@@ -0,0 +1,7 @@
+{
+  "image_token": "<image>",
+  "num_additional_image_tokens": 1,
+  "patch_size": 14,
+  "processor_class": "LlavaProcessor",
+  "vision_feature_select_strategy": "default"
+}

7b/iteration2/special_tokens_map.json

@@ -0,0 +1,31 @@
+{
+  "bos_token": {
+    "content": "<s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "</s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "image_token": "<image>",
+  "pad_token": {
+    "content": "<pad>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "unk_token": {
+    "content": "<unk>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

7b/iteration2/tokenizer.model

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9e556afd44213b6bd1be2b850ebbbd98f5481437a8021afaf58ee7fb1818d347
+size 499723

7b/iteration2/tokenizer_config.json

@@ -0,0 +1,64 @@
+{
+  "add_bos_token": true,
+  "add_eos_token": false,
+  "add_prefix_space": null,
+  "added_tokens_decoder": {
+    "0": {
+      "content": "<unk>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "1": {
+      "content": "<s>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "2": {
+      "content": "</s>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "32000": {
+      "content": "<image>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "32001": {
+      "content": "<pad>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "bos_token": "<s>",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "</s>",
+  "extra_special_tokens": {
+    "image_token": "<image>"
+  },
+  "image_token": "<image>",
+  "legacy": false,
+  "model_max_length": 1000000000000000019884624838656,
+  "pad_token": "<pad>",
+  "padding_side": "right",
+  "processor_class": "LlavaProcessor",
+  "sp_model_kwargs": {},
+  "tokenizer_class": "LlamaTokenizer",
+  "trust_remote_code": false,
+  "unk_token": "<unk>",
+  "use_default_system_prompt": false
+}

7b/iteration2/trainer_state.json

@@ -0,0 +1,2373 @@
+{
+  "best_metric": null,
+  "best_model_checkpoint": null,
+  "epoch": 4.0,
+  "eval_steps": 500,
+  "global_step": 468,
+  "is_hyper_param_search": false,
+  "is_local_process_zero": true,
+  "is_world_process_zero": true,
+  "log_history": [
+    {
+      "epoch": 0.02564102564102564,
+      "grad_norm": 1.8807990550994873,
+      "learning_rate": 2e-07,
+      "logits/chosen": 0.13671875,
+      "logits/rejected": 0.1728515625,
+      "logps/chosen": -109.5,
+      "logps/rejected": -122.5,
+      "loss": 0.6921,
+      "rewards/accuracies": 0.0416666679084301,
+      "rewards/chosen": 0.00104522705078125,
+      "rewards/margins": -0.0031280517578125,
+      "rewards/rejected": 0.004180908203125,
+      "step": 3
+    },
+    {
+      "epoch": 0.05128205128205128,
+      "grad_norm": 1.8764904737472534,
+      "learning_rate": 4e-07,
+      "logits/chosen": 0.29296875,
+      "logits/rejected": 0.345703125,
+      "logps/chosen": -106.0,
+      "logps/rejected": -128.0,
+      "loss": 0.6943,
+      "rewards/accuracies": 0.375,
+      "rewards/chosen": -0.000518798828125,
+      "rewards/margins": -0.007049560546875,
+      "rewards/rejected": 0.006500244140625,
+      "step": 6
+    },
+    {
+      "epoch": 0.07692307692307693,
+      "grad_norm": 1.893288493156433,
+      "learning_rate": 6e-07,
+      "logits/chosen": 0.1279296875,
+      "logits/rejected": 0.1923828125,
+      "logps/chosen": -99.0,
+      "logps/rejected": -110.0,
+      "loss": 0.6906,
+      "rewards/accuracies": 0.2916666567325592,
+      "rewards/chosen": 0.0036468505859375,
+      "rewards/margins": 0.00494384765625,
+      "rewards/rejected": -0.0012969970703125,
+      "step": 9
+    },
+    {
+      "epoch": 0.10256410256410256,
+      "grad_norm": 1.7804158926010132,
+      "learning_rate": 8e-07,
+      "logits/chosen": 0.28515625,
+      "logits/rejected": 0.302734375,
+      "logps/chosen": -119.0,
+      "logps/rejected": -128.0,
+      "loss": 0.6932,
+      "rewards/accuracies": 0.1666666716337204,
+      "rewards/chosen": -0.0078125,
+      "rewards/margins": -0.01214599609375,
+      "rewards/rejected": 0.004302978515625,
+      "step": 12
+    },
+    {
+      "epoch": 0.1282051282051282,
+      "grad_norm": 1.8483916521072388,
+      "learning_rate": 1e-06,
+      "logits/chosen": 0.2392578125,
+      "logits/rejected": 0.2021484375,
+      "logps/chosen": -99.5,
+      "logps/rejected": -119.5,
+      "loss": 0.6914,
+      "rewards/accuracies": 0.25,
+      "rewards/chosen": 0.0026092529296875,
+      "rewards/margins": -0.0130615234375,
+      "rewards/rejected": 0.015625,
+      "step": 15
+    },
+    {
+      "epoch": 0.15384615384615385,
+      "grad_norm": 1.9503748416900635,
+      "learning_rate": 1.2e-06,
+      "logits/chosen": 0.197265625,
+      "logits/rejected": 0.1669921875,
+      "logps/chosen": -122.0,
+      "logps/rejected": -141.0,
+      "loss": 0.6929,
+      "rewards/accuracies": 0.2916666567325592,
+      "rewards/chosen": 0.00653076171875,
+      "rewards/margins": 0.01171875,
+      "rewards/rejected": -0.005218505859375,
+      "step": 18
+    },
+    {
+      "epoch": 0.1794871794871795,
+      "grad_norm": 1.7854230403900146,
+      "learning_rate": 1.4e-06,
+      "logits/chosen": 0.208984375,
+      "logits/rejected": 0.25,
+      "logps/chosen": -106.0,
+      "logps/rejected": -108.5,
+      "loss": 0.6872,
+      "rewards/accuracies": 0.375,
+      "rewards/chosen": -0.00182342529296875,
+      "rewards/margins": 0.01092529296875,
+      "rewards/rejected": -0.01275634765625,
+      "step": 21
+    },
+    {
+      "epoch": 0.20512820512820512,
+      "grad_norm": 2.020820379257202,
+      "learning_rate": 1.6e-06,
+      "logits/chosen": 0.294921875,
+      "logits/rejected": 0.345703125,
+      "logps/chosen": -120.0,
+      "logps/rejected": -123.0,
+      "loss": 0.6888,
+      "rewards/accuracies": 0.3333333432674408,
+      "rewards/chosen": -0.007171630859375,
+      "rewards/margins": -0.00194549560546875,
+      "rewards/rejected": -0.00518798828125,
+      "step": 24
+    },
+    {
+      "epoch": 0.23076923076923078,
+      "grad_norm": 2.045447826385498,
+      "learning_rate": 1.8e-06,
+      "logits/chosen": 0.20703125,
+      "logits/rejected": 0.271484375,
+      "logps/chosen": -113.0,
+      "logps/rejected": -126.0,
+      "loss": 0.6945,
+      "rewards/accuracies": 0.25,
+      "rewards/chosen": -0.00885009765625,
+      "rewards/margins": -0.00469970703125,
+      "rewards/rejected": -0.004180908203125,
+      "step": 27
+    },
+    {
+      "epoch": 0.2564102564102564,
+      "grad_norm": 2.0050930976867676,
+      "learning_rate": 2e-06,
+      "logits/chosen": 0.19921875,
+      "logits/rejected": 0.27734375,
+      "logps/chosen": -122.5,
+      "logps/rejected": -145.0,
+      "loss": 0.6925,
+      "rewards/accuracies": 0.2083333283662796,
+      "rewards/chosen": -0.004180908203125,
+      "rewards/margins": -0.00311279296875,
+      "rewards/rejected": -0.00103759765625,
+      "step": 30
+    },
+    {
+      "epoch": 0.28205128205128205,
+      "grad_norm": 2.0118536949157715,
+      "learning_rate": 1.9998558164028463e-06,
+      "logits/chosen": 0.1953125,
+      "logits/rejected": 0.1748046875,
+      "logps/chosen": -124.0,
+      "logps/rejected": -143.0,
+      "loss": 0.6917,
+      "rewards/accuracies": 0.2916666567325592,
+      "rewards/chosen": -0.0020904541015625,
+      "rewards/margins": -0.004058837890625,
+      "rewards/rejected": 0.0019683837890625,
+      "step": 33
+    },
+    {
+      "epoch": 0.3076923076923077,
+      "grad_norm": 1.921610951423645,
+      "learning_rate": 1.9994233071892054e-06,
+      "logits/chosen": 0.142578125,
+      "logits/rejected": 0.1748046875,
+      "logps/chosen": -100.5,
+      "logps/rejected": -126.0,
+      "loss": 0.6862,
+      "rewards/accuracies": 0.4166666567325592,
+      "rewards/chosen": 0.0059814453125,
+      "rewards/margins": 0.0289306640625,
+      "rewards/rejected": -0.02294921875,
+      "step": 36
+    },
+    {
+      "epoch": 0.3333333333333333,
+      "grad_norm": 1.9476183652877808,
+      "learning_rate": 1.998702597080545e-06,
+      "logits/chosen": 0.1533203125,
+      "logits/rejected": 0.1689453125,
+      "logps/chosen": -134.0,
+      "logps/rejected": -134.0,
+      "loss": 0.687,
+      "rewards/accuracies": 0.2916666567325592,
+      "rewards/chosen": -0.0093994140625,
+      "rewards/margins": 0.0093994140625,
+      "rewards/rejected": -0.018798828125,
+      "step": 39
+    },
+    {
+      "epoch": 0.358974358974359,
+      "grad_norm": 2.0101871490478516,
+      "learning_rate": 1.997693893906017e-06,
+      "logits/chosen": 0.314453125,
+      "logits/rejected": 0.30078125,
+      "logps/chosen": -112.0,
+      "logps/rejected": -112.0,
+      "loss": 0.6868,
+      "rewards/accuracies": 0.375,
+      "rewards/chosen": -0.0106201171875,
+      "rewards/margins": 0.0013427734375,
+      "rewards/rejected": -0.01202392578125,
+      "step": 42
+    },
+    {
+      "epoch": 0.38461538461538464,
+      "grad_norm": 1.790059208869934,
+      "learning_rate": 1.996397488542526e-06,
+      "logits/chosen": 0.189453125,
+      "logits/rejected": 0.251953125,
+      "logps/chosen": -128.0,
+      "logps/rejected": -140.0,
+      "loss": 0.6833,
+      "rewards/accuracies": 0.4583333432674408,
+      "rewards/chosen": -0.0096435546875,
+      "rewards/margins": 0.015380859375,
+      "rewards/rejected": -0.0250244140625,
+      "step": 45
+    },
+    {
+      "epoch": 0.41025641025641024,
+      "grad_norm": 1.9502273797988892,
+      "learning_rate": 1.99481375483085e-06,
+      "logits/chosen": 0.267578125,
+      "logits/rejected": 0.2578125,
+      "logps/chosen": -105.5,
+      "logps/rejected": -108.0,
+      "loss": 0.6838,
+      "rewards/accuracies": 0.375,
+      "rewards/chosen": -0.01458740234375,
+      "rewards/margins": 0.01141357421875,
+      "rewards/rejected": -0.026123046875,
+      "step": 48
+    },
+    {
+      "epoch": 0.4358974358974359,
+      "grad_norm": 2.0794284343719482,
+      "learning_rate": 1.992943149467835e-06,
+      "logits/chosen": 0.10986328125,
+      "logits/rejected": 0.205078125,
+      "logps/chosen": -121.0,
+      "logps/rejected": -128.0,
+      "loss": 0.6795,
+      "rewards/accuracies": 0.5,
+      "rewards/chosen": 0.0,
+      "rewards/margins": 0.0291748046875,
+      "rewards/rejected": -0.029296875,
+      "step": 51
+    },
+    {
+      "epoch": 0.46153846153846156,
+      "grad_norm": 1.9881861209869385,
+      "learning_rate": 1.9907862118747023e-06,
+      "logits/chosen": 0.1796875,
+      "logits/rejected": 0.1865234375,
+      "logps/chosen": -130.0,
+      "logps/rejected": -132.0,
+      "loss": 0.6852,
+      "rewards/accuracies": 0.4166666567325592,
+      "rewards/chosen": -0.0022125244140625,
+      "rewards/margins": 0.0228271484375,
+      "rewards/rejected": -0.0250244140625,
+      "step": 54
+    },
+    {
+      "epoch": 0.48717948717948717,
+      "grad_norm": 1.941139817237854,
+      "learning_rate": 1.988343564041492e-06,
+      "logits/chosen": 0.2080078125,
+      "logits/rejected": 0.24609375,
+      "logps/chosen": -116.0,
+      "logps/rejected": -131.0,
+      "loss": 0.6751,
+      "rewards/accuracies": 0.5833333134651184,
+      "rewards/chosen": 0.023681640625,
+      "rewards/margins": 0.0615234375,
+      "rewards/rejected": -0.03759765625,
+      "step": 57
+    },
+    {
+      "epoch": 0.5128205128205128,
+      "grad_norm": 1.9498776197433472,
+      "learning_rate": 1.9856159103477083e-06,
+      "logits/chosen": 0.2041015625,
+      "logits/rejected": 0.193359375,
+      "logps/chosen": -105.5,
+      "logps/rejected": -110.0,
+      "loss": 0.6815,
+      "rewards/accuracies": 0.4583333432674408,
+      "rewards/chosen": -0.006744384765625,
+      "rewards/margins": 0.01123046875,
+      "rewards/rejected": -0.0179443359375,
+      "step": 60
+    },
+    {
+      "epoch": 0.5384615384615384,
+      "grad_norm": 2.001382827758789,
+      "learning_rate": 1.9826040373591932e-06,
+      "logits/chosen": 0.25,
+      "logits/rejected": 0.27734375,
+      "logps/chosen": -91.0,
+      "logps/rejected": -98.5,
+      "loss": 0.6746,
+      "rewards/accuracies": 0.5416666865348816,
+      "rewards/chosen": 0.01214599609375,
+      "rewards/margins": 0.03857421875,
+      "rewards/rejected": -0.0263671875,
+      "step": 63
+    },
+    {
+      "epoch": 0.5641025641025641,
+      "grad_norm": 2.089869737625122,
+      "learning_rate": 1.97930881360131e-06,
+      "logits/chosen": 0.1513671875,
+      "logits/rejected": 0.1220703125,
+      "logps/chosen": -126.5,
+      "logps/rejected": -130.0,
+      "loss": 0.6694,
+      "rewards/accuracies": 0.5416666865348816,
+      "rewards/chosen": -0.00156402587890625,
+      "rewards/margins": 0.048583984375,
+      "rewards/rejected": -0.050048828125,
+      "step": 66
+    },
+    {
+      "epoch": 0.5897435897435898,
+      "grad_norm": 1.9758906364440918,
+      "learning_rate": 1.9757311893084885e-06,
+      "logits/chosen": 0.298828125,
+      "logits/rejected": 0.203125,
+      "logps/chosen": -102.5,
+      "logps/rejected": -115.0,
+      "loss": 0.6667,
+      "rewards/accuracies": 0.5,
+      "rewards/chosen": -0.027099609375,
+      "rewards/margins": 0.0250244140625,
+      "rewards/rejected": -0.05224609375,
+      "step": 69
+    },
+    {
+      "epoch": 0.6153846153846154,
+      "grad_norm": 1.9816081523895264,
+      "learning_rate": 1.971872196150208e-06,
+      "logits/chosen": 0.259765625,
+      "logits/rejected": 0.328125,
+      "logps/chosen": -87.5,
+      "logps/rejected": -97.0,
+      "loss": 0.6711,
+      "rewards/accuracies": 0.5416666865348816,
+      "rewards/chosen": -0.00885009765625,
+      "rewards/margins": 0.035400390625,
+      "rewards/rejected": -0.04443359375,
+      "step": 72
+    },
+    {
+      "epoch": 0.6410256410256411,
+      "grad_norm": 1.8845876455307007,
+      "learning_rate": 1.967732946933499e-06,
+      "logits/chosen": 0.2373046875,
+      "logits/rejected": 0.23828125,
+      "logps/chosen": -117.0,
+      "logps/rejected": -110.5,
+      "loss": 0.6676,
+      "rewards/accuracies": 0.625,
+      "rewards/chosen": -0.0025787353515625,
+      "rewards/margins": 0.057373046875,
+      "rewards/rejected": -0.06005859375,
+      "step": 75
+    },
+    {
+      "epoch": 0.6666666666666666,
+      "grad_norm": 1.8938732147216797,
+      "learning_rate": 1.963314635282044e-06,
+      "logits/chosen": 0.263671875,
+      "logits/rejected": 0.2294921875,
+      "logps/chosen": -108.5,
+      "logps/rejected": -120.5,
+      "loss": 0.6626,
+      "rewards/accuracies": 0.625,
+      "rewards/chosen": -0.01300048828125,
+      "rewards/margins": 0.0634765625,
+      "rewards/rejected": -0.07666015625,
+      "step": 78
+    },
+    {
+      "epoch": 0.6923076923076923,
+      "grad_norm": 1.905861258506775,
+      "learning_rate": 1.9586185352919775e-06,
+      "logits/chosen": 0.181640625,
+      "logits/rejected": 0.275390625,
+      "logps/chosen": -113.5,
+      "logps/rejected": -114.0,
+      "loss": 0.6603,
+      "rewards/accuracies": 0.4583333432674408,
+      "rewards/chosen": 0.00830078125,
+      "rewards/margins": 0.06103515625,
+      "rewards/rejected": -0.052734375,
+      "step": 81
+    },
+    {
+      "epoch": 0.717948717948718,
+      "grad_norm": 2.1271400451660156,
+      "learning_rate": 1.9536460011644787e-06,
+      "logits/chosen": 0.208984375,
+      "logits/rejected": 0.2373046875,
+      "logps/chosen": -112.0,
+      "logps/rejected": -122.0,
+      "loss": 0.6602,
+      "rewards/accuracies": 0.5416666865348816,
+      "rewards/chosen": -0.007568359375,
+      "rewards/margins": 0.036865234375,
+      "rewards/rejected": -0.04443359375,
+      "step": 84
+    },
+    {
+      "epoch": 0.7435897435897436,
+      "grad_norm": 1.9534716606140137,
+      "learning_rate": 1.9483984668152616e-06,
+      "logits/chosen": 0.1650390625,
+      "logits/rejected": 0.240234375,
+      "logps/chosen": -110.0,
+      "logps/rejected": -102.0,
+      "loss": 0.6654,
+      "rewards/accuracies": 0.5833333134651184,
+      "rewards/chosen": -0.0137939453125,
+      "rewards/margins": 0.043212890625,
+      "rewards/rejected": -0.05712890625,
+      "step": 87
+    },
+    {
+      "epoch": 0.7692307692307693,
+      "grad_norm": 2.09041166305542,
+      "learning_rate": 1.942877445461084e-06,
+      "logits/chosen": 0.1435546875,
+      "logits/rejected": 0.1845703125,
+      "logps/chosen": -128.0,
+      "logps/rejected": -136.0,
+      "loss": 0.6473,
+      "rewards/accuracies": 0.875,
+      "rewards/chosen": 0.000507354736328125,
+      "rewards/margins": 0.09765625,
+      "rewards/rejected": -0.09716796875,
+      "step": 90
+    },
+    {
+      "epoch": 0.7948717948717948,
+      "grad_norm": 2.079911708831787,
+      "learning_rate": 1.9370845291833836e-06,
+      "logits/chosen": 0.1572265625,
+      "logits/rejected": 0.2158203125,
+      "logps/chosen": -109.0,
+      "logps/rejected": -123.0,
+      "loss": 0.6346,
+      "rewards/accuracies": 0.7083333134651184,
+      "rewards/chosen": -0.0223388671875,
+      "rewards/margins": 0.109375,
+      "rewards/rejected": -0.1318359375,
+      "step": 93
+    },
+    {
+      "epoch": 0.8205128205128205,
+      "grad_norm": 1.9027904272079468,
+      "learning_rate": 1.9310213884691736e-06,
+      "logits/chosen": 0.130859375,
+      "logits/rejected": 0.19140625,
+      "logps/chosen": -135.0,
+      "logps/rejected": -147.0,
+      "loss": 0.6463,
+      "rewards/accuracies": 0.5416666865348816,
+      "rewards/chosen": -0.03271484375,
+      "rewards/margins": 0.11474609375,
+      "rewards/rejected": -0.1474609375,
+      "step": 96
+    },
+    {
+      "epoch": 0.8461538461538461,
+      "grad_norm": 2.1986587047576904,
+      "learning_rate": 1.924689771729331e-06,
+      "logits/chosen": 0.224609375,
+      "logits/rejected": 0.1943359375,
+      "logps/chosen": -96.5,
+      "logps/rejected": -108.0,
+      "loss": 0.634,
+      "rewards/accuracies": 0.625,
+      "rewards/chosen": -0.0361328125,
+      "rewards/margins": 0.07275390625,
+      "rewards/rejected": -0.10888671875,
+      "step": 99
+    },
+    {
+      "epoch": 0.8717948717948718,
+      "grad_norm": 1.9795209169387817,
+      "learning_rate": 1.918091504794411e-06,
+      "logits/chosen": 0.251953125,
+      "logits/rejected": 0.2177734375,
+      "logps/chosen": -125.5,
+      "logps/rejected": -132.0,
+      "loss": 0.6447,
+      "rewards/accuracies": 0.75,
+      "rewards/chosen": -0.0240478515625,
+      "rewards/margins": 0.1845703125,
+      "rewards/rejected": -0.208984375,
+      "step": 102
+    },
+    {
+      "epoch": 0.8974358974358975,
+      "grad_norm": 2.002471923828125,
+      "learning_rate": 1.9112284903881357e-06,
+      "logits/chosen": 0.228515625,
+      "logits/rejected": 0.25,
+      "logps/chosen": -105.5,
+      "logps/rejected": -117.0,
+      "loss": 0.6333,
+      "rewards/accuracies": 0.7083333134651184,
+      "rewards/chosen": -0.0177001953125,
+      "rewards/margins": 0.126953125,
+      "rewards/rejected": -0.14453125,
+      "step": 105
+    },
+    {
+      "epoch": 0.9230769230769231,
+      "grad_norm": 1.976642370223999,
+      "learning_rate": 1.9041027075787146e-06,
+      "logits/chosen": 0.216796875,
+      "logits/rejected": 0.185546875,
+      "logps/chosen": -111.0,
+      "logps/rejected": -133.0,
+      "loss": 0.6283,
+      "rewards/accuracies": 0.625,
+      "rewards/chosen": -0.06884765625,
+      "rewards/margins": 0.1318359375,
+      "rewards/rejected": -0.201171875,
+      "step": 108
+    },
+    {
+      "epoch": 0.9487179487179487,
+      "grad_norm": 2.043111562728882,
+      "learning_rate": 1.8967162112081435e-06,
+      "logits/chosen": 0.1640625,
+      "logits/rejected": 0.1767578125,
+      "logps/chosen": -147.0,
+      "logps/rejected": -144.0,
+      "loss": 0.6125,
+      "rewards/accuracies": 0.75,
+      "rewards/chosen": -0.06494140625,
+      "rewards/margins": 0.2373046875,
+      "rewards/rejected": -0.302734375,
+      "step": 111
+    },
+    {
+      "epoch": 0.9743589743589743,
+      "grad_norm": 2.0380313396453857,
+      "learning_rate": 1.8890711312996568e-06,
+      "logits/chosen": 0.353515625,
+      "logits/rejected": 0.353515625,
+      "logps/chosen": -110.0,
+      "logps/rejected": -113.0,
+      "loss": 0.6395,
+      "rewards/accuracies": 0.4583333432674408,
+      "rewards/chosen": -0.091796875,
+      "rewards/margins": 0.07666015625,
+      "rewards/rejected": -0.16796875,
+      "step": 114
+    },
+    {
+      "epoch": 1.0,
+      "grad_norm": 2.0977935791015625,
+      "learning_rate": 1.881169672443498e-06,
+      "logits/chosen": 0.2275390625,
+      "logits/rejected": 0.2373046875,
+      "logps/chosen": -104.0,
+      "logps/rejected": -128.0,
+      "loss": 0.6076,
+      "rewards/accuracies": 0.7083333134651184,
+      "rewards/chosen": -0.078125,
+      "rewards/margins": 0.2041015625,
+      "rewards/rejected": -0.283203125,
+      "step": 117
+    },
+    {
+      "epoch": 1.0256410256410255,
+      "grad_norm": 2.0318846702575684,
+      "learning_rate": 1.873014113161188e-06,
+      "logits/chosen": 0.2060546875,
+      "logits/rejected": 0.1337890625,
+      "logps/chosen": -114.5,
+      "logps/rejected": -120.0,
+      "loss": 0.5905,
+      "rewards/accuracies": 0.7083333134651184,
+      "rewards/chosen": -0.1025390625,
+      "rewards/margins": 0.23828125,
+      "rewards/rejected": -0.341796875,
+      "step": 120
+    },
+    {
+      "epoch": 1.0512820512820513,
+      "grad_norm": 1.9920344352722168,
+      "learning_rate": 1.8646068052484753e-06,
+      "logits/chosen": 0.1767578125,
+      "logits/rejected": 0.1533203125,
+      "logps/chosen": -120.5,
+      "logps/rejected": -128.0,
+      "loss": 0.6056,
+      "rewards/accuracies": 0.8333333134651184,
+      "rewards/chosen": -0.0194091796875,
+      "rewards/margins": 0.248046875,
+      "rewards/rejected": -0.267578125,
+      "step": 123
+    },
+    {
+      "epoch": 1.0769230769230769,
+      "grad_norm": 1.8763842582702637,
+      "learning_rate": 1.8559501730971543e-06,
+      "logits/chosen": 0.220703125,
+      "logits/rejected": 0.298828125,
+      "logps/chosen": -108.0,
+      "logps/rejected": -131.0,
+      "loss": 0.5767,
+      "rewards/accuracies": 0.8333333134651184,
+      "rewards/chosen": -0.11181640625,
+      "rewards/margins": 0.361328125,
+      "rewards/rejected": -0.47265625,
+      "step": 126
+    },
+    {
+      "epoch": 1.1025641025641026,
+      "grad_norm": 1.9118515253067017,
+      "learning_rate": 1.8470467129959508e-06,
+      "logits/chosen": 0.2431640625,
+      "logits/rejected": 0.236328125,
+      "logps/chosen": -98.5,
+      "logps/rejected": -120.5,
+      "loss": 0.578,
+      "rewards/accuracies": 0.875,
+      "rewards/chosen": -0.06982421875,
+      "rewards/margins": 0.404296875,
+      "rewards/rejected": -0.47265625,
+      "step": 129
+    },
+    {
+      "epoch": 1.1282051282051282,
+      "grad_norm": 2.0091183185577393,
+      "learning_rate": 1.8378989924106735e-06,
+      "logits/chosen": 0.1865234375,
+      "logits/rejected": 0.25,
+      "logps/chosen": -111.5,
+      "logps/rejected": -124.0,
+      "loss": 0.5826,
+      "rewards/accuracies": 0.7083333134651184,
+      "rewards/chosen": -0.10302734375,
+      "rewards/margins": 0.28515625,
+      "rewards/rejected": -0.388671875,
+      "step": 132
+    },
+    {
+      "epoch": 1.1538461538461537,
+      "grad_norm": 2.098787307739258,
+      "learning_rate": 1.828509649243842e-06,
+      "logits/chosen": 0.09521484375,
+      "logits/rejected": 0.091796875,
+      "logps/chosen": -127.0,
+      "logps/rejected": -135.0,
+      "loss": 0.5577,
+      "rewards/accuracies": 0.7083333134651184,
+      "rewards/chosen": -0.0986328125,
+      "rewards/margins": 0.255859375,
+      "rewards/rejected": -0.353515625,
+      "step": 135
+    },
+    {
+      "epoch": 1.1794871794871795,
+      "grad_norm": 1.9569436311721802,
+      "learning_rate": 1.8188813910740017e-06,
+      "logits/chosen": 0.251953125,
+      "logits/rejected": 0.1982421875,
+      "logps/chosen": -111.0,
+      "logps/rejected": -140.0,
+      "loss": 0.5628,
+      "rewards/accuracies": 0.875,
+      "rewards/chosen": -0.1591796875,
+      "rewards/margins": 0.400390625,
+      "rewards/rejected": -0.55859375,
+      "step": 138
+    },
+    {
+      "epoch": 1.205128205128205,
+      "grad_norm": 2.0690457820892334,
+      "learning_rate": 1.8090169943749474e-06,
+      "logits/chosen": 0.2041015625,
+      "logits/rejected": 0.1884765625,
+      "logps/chosen": -110.5,
+      "logps/rejected": -123.0,
+      "loss": 0.5893,
+      "rewards/accuracies": 0.5833333134651184,
+      "rewards/chosen": -0.298828125,
+      "rewards/margins": 0.1181640625,
+      "rewards/rejected": -0.416015625,
+      "step": 141
+    },
+    {
+      "epoch": 1.2307692307692308,
+      "grad_norm": 2.074464797973633,
+      "learning_rate": 1.7989193037150782e-06,
+      "logits/chosen": 0.26953125,
+      "logits/rejected": 0.265625,
+      "logps/chosen": -110.0,
+      "logps/rejected": -128.0,
+      "loss": 0.5638,
+      "rewards/accuracies": 0.875,
+      "rewards/chosen": -0.2412109375,
+      "rewards/margins": 0.314453125,
+      "rewards/rejected": -0.5546875,
+      "step": 144
+    },
+    {
+      "epoch": 1.2564102564102564,
+      "grad_norm": 2.286728858947754,
+      "learning_rate": 1.788591230937119e-06,
+      "logits/chosen": 0.09619140625,
+      "logits/rejected": 0.06689453125,
+      "logps/chosen": -113.0,
+      "logps/rejected": -136.0,
+      "loss": 0.5804,
+      "rewards/accuracies": 0.75,
+      "rewards/chosen": -0.26171875,
+      "rewards/margins": 0.259765625,
+      "rewards/rejected": -0.5234375,
+      "step": 147
+    },
+    {
+      "epoch": 1.282051282051282,
+      "grad_norm": 2.0626838207244873,
+      "learning_rate": 1.7780357543184393e-06,
+      "logits/chosen": 0.1767578125,
+      "logits/rejected": 0.185546875,
+      "logps/chosen": -121.0,
+      "logps/rejected": -115.5,
+      "loss": 0.5309,
+      "rewards/accuracies": 0.75,
+      "rewards/chosen": -0.2236328125,
+      "rewards/margins": 0.314453125,
+      "rewards/rejected": -0.5390625,
+      "step": 150
+    },
+    {
+      "epoch": 1.3076923076923077,
+      "grad_norm": 2.1246135234832764,
+      "learning_rate": 1.7672559177122163e-06,
+      "logits/chosen": 0.19140625,
+      "logits/rejected": 0.1806640625,
+      "logps/chosen": -118.0,
+      "logps/rejected": -148.0,
+      "loss": 0.5572,
+      "rewards/accuracies": 0.75,
+      "rewards/chosen": -0.31640625,
+      "rewards/margins": 0.427734375,
+      "rewards/rejected": -0.7421875,
+      "step": 153
+    },
+    {
+      "epoch": 1.3333333333333333,
+      "grad_norm": 1.9632675647735596,
+      "learning_rate": 1.7562548296696873e-06,
+      "logits/chosen": 0.2138671875,
+      "logits/rejected": 0.220703125,
+      "logps/chosen": -103.0,
+      "logps/rejected": -125.5,
+      "loss": 0.5375,
+      "rewards/accuracies": 0.7083333134651184,
+      "rewards/chosen": -0.2255859375,
+      "rewards/margins": 0.353515625,
+      "rewards/rejected": -0.578125,
+      "step": 156
+    },
+    {
+      "epoch": 1.358974358974359,
+      "grad_norm": 1.9024431705474854,
+      "learning_rate": 1.745035662543745e-06,
+      "logits/chosen": 0.1787109375,
+      "logits/rejected": 0.1591796875,
+      "logps/chosen": -116.5,
+      "logps/rejected": -146.0,
+      "loss": 0.5115,
+      "rewards/accuracies": 0.9166666865348816,
+      "rewards/chosen": -0.220703125,
+      "rewards/margins": 0.58203125,
+      "rewards/rejected": -0.8046875,
+      "step": 159
+    },
+    {
+      "epoch": 1.3846153846153846,
+      "grad_norm": 2.3646910190582275,
+      "learning_rate": 1.7336016515741365e-06,
+      "logits/chosen": 0.189453125,
+      "logits/rejected": 0.1572265625,
+      "logps/chosen": -132.0,
+      "logps/rejected": -156.0,
+      "loss": 0.54,
+      "rewards/accuracies": 0.875,
+      "rewards/chosen": -0.2734375,
+      "rewards/margins": 0.64453125,
+      "rewards/rejected": -0.91796875,
+      "step": 162
+    },
+    {
+      "epoch": 1.4102564102564101,
+      "grad_norm": 2.328059434890747,
+      "learning_rate": 1.7219560939545242e-06,
+      "logits/chosen": 0.232421875,
+      "logits/rejected": 0.2294921875,
+      "logps/chosen": -96.0,
+      "logps/rejected": -109.0,
+      "loss": 0.5638,
+      "rewards/accuracies": 0.75,
+      "rewards/chosen": -0.298828125,
+      "rewards/margins": 0.41015625,
+      "rewards/rejected": -0.7109375,
+      "step": 165
+    },
+    {
+      "epoch": 1.435897435897436,
+      "grad_norm": 2.036503791809082,
+      "learning_rate": 1.7101023478816856e-06,
+      "logits/chosen": 0.1669921875,
+      "logits/rejected": 0.2060546875,
+      "logps/chosen": -132.0,
+      "logps/rejected": -131.0,
+      "loss": 0.5173,
+      "rewards/accuracies": 0.8333333134651184,
+      "rewards/chosen": -0.4140625,
+      "rewards/margins": 0.412109375,
+      "rewards/rejected": -0.82421875,
+      "step": 168
+    },
+    {
+      "epoch": 1.4615384615384617,
+      "grad_norm": 2.166957378387451,
+      "learning_rate": 1.6980438315871177e-06,
+      "logits/chosen": 0.1669921875,
+      "logits/rejected": 0.2021484375,
+      "logps/chosen": -113.0,
+      "logps/rejected": -119.5,
+      "loss": 0.5193,
+      "rewards/accuracies": 0.875,
+      "rewards/chosen": -0.26953125,
+      "rewards/margins": 0.51953125,
+      "rewards/rejected": -0.7890625,
+      "step": 171
+    },
+    {
+      "epoch": 1.4871794871794872,
+      "grad_norm": 2.383507490158081,
+      "learning_rate": 1.6857840223513313e-06,
+      "logits/chosen": 0.2060546875,
+      "logits/rejected": 0.19921875,
+      "logps/chosen": -120.0,
+      "logps/rejected": -124.0,
+      "loss": 0.5446,
+      "rewards/accuracies": 0.7916666865348816,
+      "rewards/chosen": -0.361328125,
+      "rewards/margins": 0.431640625,
+      "rewards/rejected": -0.79296875,
+      "step": 174
+    },
+    {
+      "epoch": 1.5128205128205128,
+      "grad_norm": 2.0628299713134766,
+      "learning_rate": 1.6733264555011194e-06,
+      "logits/chosen": 0.244140625,
+      "logits/rejected": 0.20703125,
+      "logps/chosen": -137.0,
+      "logps/rejected": -154.0,
+      "loss": 0.5155,
+      "rewards/accuracies": 0.875,
+      "rewards/chosen": -0.57421875,
+      "rewards/margins": 0.73828125,
+      "rewards/rejected": -1.3125,
+      "step": 177
+    },
+    {
+      "epoch": 1.5384615384615383,
+      "grad_norm": 2.3130903244018555,
+      "learning_rate": 1.6606747233900813e-06,
+      "logits/chosen": 0.2431640625,
+      "logits/rejected": 0.2451171875,
+      "logps/chosen": -96.0,
+      "logps/rejected": -111.5,
+      "loss": 0.5542,
+      "rewards/accuracies": 0.875,
+      "rewards/chosen": -0.2294921875,
+      "rewards/margins": 0.7109375,
+      "rewards/rejected": -0.9375,
+      "step": 180
+    },
+    {
+      "epoch": 1.564102564102564,
+      "grad_norm": 2.157144784927368,
+      "learning_rate": 1.64783247436271e-06,
+      "logits/chosen": 0.298828125,
+      "logits/rejected": 0.314453125,
+      "logps/chosen": -124.0,
+      "logps/rejected": -143.0,
+      "loss": 0.5016,
+      "rewards/accuracies": 0.8333333134651184,
+      "rewards/chosen": -0.408203125,
+      "rewards/margins": 0.6015625,
+      "rewards/rejected": -1.015625,
+      "step": 183
+    },
+    {
+      "epoch": 1.5897435897435899,
+      "grad_norm": 2.006909132003784,
+      "learning_rate": 1.6348034117023256e-06,
+      "logits/chosen": 0.140625,
+      "logits/rejected": 0.1650390625,
+      "logps/chosen": -109.0,
+      "logps/rejected": -137.0,
+      "loss": 0.5129,
+      "rewards/accuracies": 0.7916666865348816,
+      "rewards/chosen": -0.3984375,
+      "rewards/margins": 0.703125,
+      "rewards/rejected": -1.1015625,
+      "step": 186
+    },
+    {
+      "epoch": 1.6153846153846154,
+      "grad_norm": 2.5358331203460693,
+      "learning_rate": 1.6215912925631721e-06,
+      "logits/chosen": 0.1962890625,
+      "logits/rejected": 0.21484375,
+      "logps/chosen": -114.0,
+      "logps/rejected": -134.0,
+      "loss": 0.5205,
+      "rewards/accuracies": 0.8333333134651184,
+      "rewards/chosen": -0.478515625,
+      "rewards/margins": 0.74609375,
+      "rewards/rejected": -1.2265625,
+      "step": 189
+    },
+    {
+      "epoch": 1.641025641025641,
+      "grad_norm": 2.1718602180480957,
+      "learning_rate": 1.6081999268869763e-06,
+      "logits/chosen": 0.294921875,
+      "logits/rejected": 0.3203125,
+      "logps/chosen": -109.0,
+      "logps/rejected": -130.0,
+      "loss": 0.4935,
+      "rewards/accuracies": 0.7916666865348816,
+      "rewards/chosen": -0.5625,
+      "rewards/margins": 0.50390625,
+      "rewards/rejected": -1.0703125,
+      "step": 192
+    },
+    {
+      "epoch": 1.6666666666666665,
+      "grad_norm": 2.2296667098999023,
+      "learning_rate": 1.5946331763042866e-06,
+      "logits/chosen": 0.1357421875,
+      "logits/rejected": 0.2294921875,
+      "logps/chosen": -122.0,
+      "logps/rejected": -114.5,
+      "loss": 0.5057,
+      "rewards/accuracies": 0.75,
+      "rewards/chosen": -0.53515625,
+      "rewards/margins": 0.50390625,
+      "rewards/rejected": -1.0390625,
+      "step": 195
+    },
+    {
+      "epoch": 1.6923076923076923,
+      "grad_norm": 2.204519748687744,
+      "learning_rate": 1.580894953020904e-06,
+      "logits/chosen": 0.1435546875,
+      "logits/rejected": 0.150390625,
+      "logps/chosen": -113.5,
+      "logps/rejected": -135.0,
+      "loss": 0.5566,
+      "rewards/accuracies": 0.8333333134651184,
+      "rewards/chosen": -0.421875,
+      "rewards/margins": 0.76171875,
+      "rewards/rejected": -1.1875,
+      "step": 198
+    },
+    {
+      "epoch": 1.717948717948718,
+      "grad_norm": 2.3120129108428955,
+      "learning_rate": 1.5669892186897316e-06,
+      "logits/chosen": 0.1572265625,
+      "logits/rejected": 0.208984375,
+      "logps/chosen": -124.0,
+      "logps/rejected": -141.0,
+      "loss": 0.4966,
+      "rewards/accuracies": 0.8333333134651184,
+      "rewards/chosen": -0.53515625,
+      "rewards/margins": 1.0,
+      "rewards/rejected": -1.5390625,
+      "step": 201
+    },
+    {
+      "epoch": 1.7435897435897436,
+      "grad_norm": 2.3819029331207275,
+      "learning_rate": 1.5529199832683633e-06,
+      "logits/chosen": 0.171875,
+      "logits/rejected": 0.2236328125,
+      "logps/chosen": -128.0,
+      "logps/rejected": -150.0,
+      "loss": 0.4868,
+      "rewards/accuracies": 0.8333333134651184,
+      "rewards/chosen": -0.64453125,
+      "rewards/margins": 0.7734375,
+      "rewards/rejected": -1.421875,
+      "step": 204
+    },
+    {
+      "epoch": 1.7692307692307692,
+      "grad_norm": 2.1900620460510254,
+      "learning_rate": 1.5386913038627438e-06,
+      "logits/chosen": 0.1767578125,
+      "logits/rejected": 0.1796875,
+      "logps/chosen": -152.0,
+      "logps/rejected": -139.0,
+      "loss": 0.5002,
+      "rewards/accuracies": 0.7916666865348816,
+      "rewards/chosen": -0.875,
+      "rewards/margins": 0.69921875,
+      "rewards/rejected": -1.5703125,
+      "step": 207
+    },
+    {
+      "epoch": 1.7948717948717947,
+      "grad_norm": 2.0419483184814453,
+      "learning_rate": 1.5243072835572316e-06,
+      "logits/chosen": 0.158203125,
+      "logits/rejected": 0.1796875,
+      "logps/chosen": -128.0,
+      "logps/rejected": -138.0,
+      "loss": 0.4862,
+      "rewards/accuracies": 0.9166666865348816,
+      "rewards/chosen": -0.69140625,
+      "rewards/margins": 0.70703125,
+      "rewards/rejected": -1.3984375,
+      "step": 210
+    },
+    {
+      "epoch": 1.8205128205128205,
+      "grad_norm": 2.071697473526001,
+      "learning_rate": 1.5097720702314054e-06,
+      "logits/chosen": 0.11083984375,
+      "logits/rejected": 0.16015625,
+      "logps/chosen": -129.0,
+      "logps/rejected": -157.0,
+      "loss": 0.4742,
+      "rewards/accuracies": 0.7916666865348816,
+      "rewards/chosen": -0.66015625,
+      "rewards/margins": 1.140625,
+      "rewards/rejected": -1.8046875,
+      "step": 213
+    },
+    {
+      "epoch": 1.8461538461538463,
+      "grad_norm": 2.228717088699341,
+      "learning_rate": 1.4950898553639504e-06,
+      "logits/chosen": 0.06396484375,
+      "logits/rejected": 0.173828125,
+      "logps/chosen": -137.0,
+      "logps/rejected": -140.0,
+      "loss": 0.4845,
+      "rewards/accuracies": 0.9166666865348816,
+      "rewards/chosen": -0.69140625,
+      "rewards/margins": 0.8125,
+      "rewards/rejected": -1.5078125,
+      "step": 216
+    },
+    {
+      "epoch": 1.8717948717948718,
+      "grad_norm": 2.5235626697540283,
+      "learning_rate": 1.4802648728239743e-06,
+      "logits/chosen": 0.2421875,
+      "logits/rejected": 0.197265625,
+      "logps/chosen": -98.0,
+      "logps/rejected": -112.5,
+      "loss": 0.5188,
+      "rewards/accuracies": 0.7916666865348816,
+      "rewards/chosen": -0.63671875,
+      "rewards/margins": 0.369140625,
+      "rewards/rejected": -1.0,
+      "step": 219
+    },
+    {
+      "epoch": 1.8974358974358974,
+      "grad_norm": 2.2643330097198486,
+      "learning_rate": 1.4653013976500974e-06,
+      "logits/chosen": 0.203125,
+      "logits/rejected": 0.2080078125,
+      "logps/chosen": -143.0,
+      "logps/rejected": -159.0,
+      "loss": 0.4687,
+      "rewards/accuracies": 0.7916666865348816,
+      "rewards/chosen": -0.875,
+      "rewards/margins": 0.8984375,
+      "rewards/rejected": -1.7734375,
+      "step": 222
+    },
+    {
+      "epoch": 1.9230769230769231,
+      "grad_norm": 2.175093650817871,
+      "learning_rate": 1.4502037448176732e-06,
+      "logits/chosen": 0.138671875,
+      "logits/rejected": 0.1689453125,
+      "logps/chosen": -114.5,
+      "logps/rejected": -138.0,
+      "loss": 0.5153,
+      "rewards/accuracies": 0.7916666865348816,
+      "rewards/chosen": -0.6171875,
+      "rewards/margins": 0.8671875,
+      "rewards/rejected": -1.484375,
+      "step": 225
+    },
+    {
+      "epoch": 1.9487179487179487,
+      "grad_norm": 2.8286232948303223,
+      "learning_rate": 1.4349762679944895e-06,
+      "logits/chosen": 0.15234375,
+      "logits/rejected": 0.154296875,
+      "logps/chosen": -113.5,
+      "logps/rejected": -131.0,
+      "loss": 0.4437,
+      "rewards/accuracies": 0.75,
+      "rewards/chosen": -0.74609375,
+      "rewards/margins": 0.78515625,
+      "rewards/rejected": -1.53125,
+      "step": 228
+    },
+    {
+      "epoch": 1.9743589743589745,
+      "grad_norm": 2.7549867630004883,
+      "learning_rate": 1.419623358285314e-06,
+      "logits/chosen": 0.142578125,
+      "logits/rejected": 0.11669921875,
+      "logps/chosen": -106.0,
+      "logps/rejected": -130.0,
+      "loss": 0.4782,
+      "rewards/accuracies": 0.8333333134651184,
+      "rewards/chosen": -0.52734375,
+      "rewards/margins": 0.796875,
+      "rewards/rejected": -1.328125,
+      "step": 231
+    },
+    {
+      "epoch": 2.0,
+      "grad_norm": 1.993396520614624,
+      "learning_rate": 1.404149442965644e-06,
+      "logits/chosen": 0.234375,
+      "logits/rejected": 0.1884765625,
+      "logps/chosen": -107.0,
+      "logps/rejected": -129.0,
+      "loss": 0.4322,
+      "rewards/accuracies": 0.7916666865348816,
+      "rewards/chosen": -0.80078125,
+      "rewards/margins": 0.89453125,
+      "rewards/rejected": -1.6953125,
+      "step": 234
+    },
+    {
+      "epoch": 2.0256410256410255,
+      "grad_norm": 2.6139297485351562,
+      "learning_rate": 1.3885589842050252e-06,
+      "logits/chosen": 0.314453125,
+      "logits/rejected": 0.267578125,
+      "logps/chosen": -94.5,
+      "logps/rejected": -105.0,
+      "loss": 0.479,
+      "rewards/accuracies": 0.7916666865348816,
+      "rewards/chosen": -0.7265625,
+      "rewards/margins": 0.71875,
+      "rewards/rejected": -1.4453125,
+      "step": 237
+    },
+    {
+      "epoch": 2.051282051282051,
+      "grad_norm": 1.9830079078674316,
+      "learning_rate": 1.3728564777803086e-06,
+      "logits/chosen": 0.126953125,
+      "logits/rejected": 0.1376953125,
+      "logps/chosen": -111.5,
+      "logps/rejected": -123.5,
+      "loss": 0.4288,
+      "rewards/accuracies": 0.7083333134651184,
+      "rewards/chosen": -0.76953125,
+      "rewards/margins": 0.7265625,
+      "rewards/rejected": -1.5,
+      "step": 240
+    },
+    {
+      "epoch": 2.076923076923077,
+      "grad_norm": 2.3474740982055664,
+      "learning_rate": 1.3570464517792152e-06,
+      "logits/chosen": 0.1708984375,
+      "logits/rejected": 0.1572265625,
+      "logps/chosen": -106.0,
+      "logps/rejected": -145.0,
+      "loss": 0.4326,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": -0.73046875,
+      "rewards/margins": 1.4921875,
+      "rewards/rejected": -2.21875,
+      "step": 243
+    },
+    {
+      "epoch": 2.1025641025641026,
+      "grad_norm": 2.3264522552490234,
+      "learning_rate": 1.3411334652945848e-06,
+      "logits/chosen": 0.236328125,
+      "logits/rejected": 0.265625,
+      "logps/chosen": -115.0,
+      "logps/rejected": -131.0,
+      "loss": 0.4788,
+      "rewards/accuracies": 0.75,
+      "rewards/chosen": -1.03125,
+      "rewards/margins": 0.5234375,
+      "rewards/rejected": -1.546875,
+      "step": 246
+    },
+    {
+      "epoch": 2.128205128205128,
+      "grad_norm": 2.1243481636047363,
+      "learning_rate": 1.3251221071096834e-06,
+      "logits/chosen": 0.2255859375,
+      "logits/rejected": 0.1943359375,
+      "logps/chosen": -117.0,
+      "logps/rejected": -120.0,
+      "loss": 0.4144,
+      "rewards/accuracies": 0.875,
+      "rewards/chosen": -0.83984375,
+      "rewards/margins": 0.65234375,
+      "rewards/rejected": -1.4921875,
+      "step": 249
+    },
+    {
+      "epoch": 2.1538461538461537,
+      "grad_norm": 2.6467950344085693,
+      "learning_rate": 1.3090169943749473e-06,
+      "logits/chosen": 0.22265625,
+      "logits/rejected": 0.25390625,
+      "logps/chosen": -117.5,
+      "logps/rejected": -138.0,
+      "loss": 0.4197,
+      "rewards/accuracies": 0.8333333134651184,
+      "rewards/chosen": -0.81640625,
+      "rewards/margins": 0.9765625,
+      "rewards/rejected": -1.7890625,
+      "step": 252
+    },
+    {
+      "epoch": 2.1794871794871793,
+      "grad_norm": 2.3144333362579346,
+      "learning_rate": 1.2928227712765502e-06,
+      "logits/chosen": 0.1650390625,
+      "logits/rejected": 0.1767578125,
+      "logps/chosen": -120.5,
+      "logps/rejected": -131.0,
+      "loss": 0.4023,
+      "rewards/accuracies": 0.7916666865348816,
+      "rewards/chosen": -0.96484375,
+      "rewards/margins": 0.93359375,
+      "rewards/rejected": -1.8984375,
+      "step": 255
+    },
+    {
+      "epoch": 2.2051282051282053,
+      "grad_norm": 2.361914873123169,
+      "learning_rate": 1.276544107697171e-06,
+      "logits/chosen": 0.236328125,
+      "logits/rejected": 0.197265625,
+      "logps/chosen": -138.0,
+      "logps/rejected": -149.0,
+      "loss": 0.4806,
+      "rewards/accuracies": 0.7916666865348816,
+      "rewards/chosen": -1.1171875,
+      "rewards/margins": 0.92578125,
+      "rewards/rejected": -2.046875,
+      "step": 258
+    },
+    {
+      "epoch": 2.230769230769231,
+      "grad_norm": 2.1916277408599854,
+      "learning_rate": 1.260185697869353e-06,
+      "logits/chosen": 0.173828125,
+      "logits/rejected": 0.2060546875,
+      "logps/chosen": -125.0,
+      "logps/rejected": -133.0,
+      "loss": 0.4517,
+      "rewards/accuracies": 0.875,
+      "rewards/chosen": -0.84375,
+      "rewards/margins": 1.0859375,
+      "rewards/rejected": -1.9296875,
+      "step": 261
+    },
+    {
+      "epoch": 2.2564102564102564,
+      "grad_norm": 2.8083648681640625,
+      "learning_rate": 1.2437522590218416e-06,
+      "logits/chosen": 0.20703125,
+      "logits/rejected": 0.2021484375,
+      "logps/chosen": -109.0,
+      "logps/rejected": -131.0,
+      "loss": 0.4198,
+      "rewards/accuracies": 0.875,
+      "rewards/chosen": -0.98046875,
+      "rewards/margins": 1.03125,
+      "rewards/rejected": -2.015625,
+      "step": 264
+    },
+    {
+      "epoch": 2.282051282051282,
+      "grad_norm": 3.0645532608032227,
+      "learning_rate": 1.22724853001929e-06,
+      "logits/chosen": 0.1259765625,
+      "logits/rejected": 0.1630859375,
+      "logps/chosen": -113.0,
+      "logps/rejected": -129.0,
+      "loss": 0.4473,
+      "rewards/accuracies": 0.875,
+      "rewards/chosen": -0.84765625,
+      "rewards/margins": 1.0703125,
+      "rewards/rejected": -1.921875,
+      "step": 267
+    },
+    {
+      "epoch": 2.3076923076923075,
+      "grad_norm": 2.734609842300415,
+      "learning_rate": 1.2106792699957262e-06,
+      "logits/chosen": 0.1328125,
+      "logits/rejected": 0.1103515625,
+      "logps/chosen": -137.0,
+      "logps/rejected": -157.0,
+      "loss": 0.4295,
+      "rewards/accuracies": 0.75,
+      "rewards/chosen": -1.25,
+      "rewards/margins": 1.1171875,
+      "rewards/rejected": -2.359375,
+      "step": 270
+    },
+    {
+      "epoch": 2.3333333333333335,
+      "grad_norm": 3.1695902347564697,
+      "learning_rate": 1.1940492569821752e-06,
+      "logits/chosen": 0.171875,
+      "logits/rejected": 0.2431640625,
+      "logps/chosen": -129.0,
+      "logps/rejected": -162.0,
+      "loss": 0.4173,
+      "rewards/accuracies": 0.8333333134651184,
+      "rewards/chosen": -1.15625,
+      "rewards/margins": 1.453125,
+      "rewards/rejected": -2.625,
+      "step": 273
+    },
+    {
+      "epoch": 2.358974358974359,
+      "grad_norm": 2.7723398208618164,
+      "learning_rate": 1.1773632865288308e-06,
+      "logits/chosen": 0.1962890625,
+      "logits/rejected": 0.130859375,
+      "logps/chosen": -126.0,
+      "logps/rejected": -148.0,
+      "loss": 0.4664,
+      "rewards/accuracies": 0.875,
+      "rewards/chosen": -1.09375,
+      "rewards/margins": 1.296875,
+      "rewards/rejected": -2.375,
+      "step": 276
+    },
+    {
+      "epoch": 2.3846153846153846,
+      "grad_norm": 2.630082607269287,
+      "learning_rate": 1.160626170322177e-06,
+      "logits/chosen": 0.09912109375,
+      "logits/rejected": 0.134765625,
+      "logps/chosen": -119.0,
+      "logps/rejected": -147.0,
+      "loss": 0.4221,
+      "rewards/accuracies": 0.9166666865348816,
+      "rewards/chosen": -0.91015625,
+      "rewards/margins": 1.1875,
+      "rewards/rejected": -2.09375,
+      "step": 279
+    },
+    {
+      "epoch": 2.41025641025641,
+      "grad_norm": 2.4502968788146973,
+      "learning_rate": 1.1438427347974552e-06,
+      "logits/chosen": 0.17578125,
+      "logits/rejected": 0.1787109375,
+      "logps/chosen": -130.0,
+      "logps/rejected": -167.0,
+      "loss": 0.3601,
+      "rewards/accuracies": 0.8333333134651184,
+      "rewards/chosen": -1.03125,
+      "rewards/margins": 1.40625,
+      "rewards/rejected": -2.4375,
+      "step": 282
+    },
+    {
+      "epoch": 2.435897435897436,
+      "grad_norm": 2.8177709579467773,
+      "learning_rate": 1.1270178197468786e-06,
+      "logits/chosen": 0.1044921875,
+      "logits/rejected": 0.0966796875,
+      "logps/chosen": -122.5,
+      "logps/rejected": -147.0,
+      "loss": 0.4305,
+      "rewards/accuracies": 0.875,
+      "rewards/chosen": -0.9765625,
+      "rewards/margins": 1.265625,
+      "rewards/rejected": -2.25,
+      "step": 285
+    },
+    {
+      "epoch": 2.4615384615384617,
+      "grad_norm": 1.9717708826065063,
+      "learning_rate": 1.1101562769239945e-06,
+      "logits/chosen": 0.20703125,
+      "logits/rejected": 0.2021484375,
+      "logps/chosen": -115.5,
+      "logps/rejected": -141.0,
+      "loss": 0.3989,
+      "rewards/accuracies": 0.7916666865348816,
+      "rewards/chosen": -1.0234375,
+      "rewards/margins": 1.375,
+      "rewards/rejected": -2.390625,
+      "step": 288
+    },
+    {
+      "epoch": 2.4871794871794872,
+      "grad_norm": 2.79207181930542,
+      "learning_rate": 1.0932629686445985e-06,
+      "logits/chosen": 0.08056640625,
+      "logits/rejected": 0.07568359375,
+      "logps/chosen": -99.0,
+      "logps/rejected": -112.5,
+      "loss": 0.4288,
+      "rewards/accuracies": 0.8333333134651184,
+      "rewards/chosen": -0.796875,
+      "rewards/margins": 1.0234375,
+      "rewards/rejected": -1.828125,
+      "step": 291
+    },
+    {
+      "epoch": 2.5128205128205128,
+      "grad_norm": 2.2847957611083984,
+      "learning_rate": 1.0763427663846013e-06,
+      "logits/chosen": 0.07373046875,
+      "logits/rejected": 0.06494140625,
+      "logps/chosen": -118.5,
+      "logps/rejected": -153.0,
+      "loss": 0.3853,
+      "rewards/accuracies": 0.875,
+      "rewards/chosen": -0.796875,
+      "rewards/margins": 1.484375,
+      "rewards/rejected": -2.28125,
+      "step": 294
+    },
+    {
+      "epoch": 2.5384615384615383,
+      "grad_norm": 2.584763288497925,
+      "learning_rate": 1.0594005493752567e-06,
+      "logits/chosen": 0.1416015625,
+      "logits/rejected": 0.099609375,
+      "logps/chosen": -103.0,
+      "logps/rejected": -112.0,
+      "loss": 0.4753,
+      "rewards/accuracies": 0.7916666865348816,
+      "rewards/chosen": -0.84375,
+      "rewards/margins": 0.8359375,
+      "rewards/rejected": -1.6796875,
+      "step": 297
+    },
+    {
+      "epoch": 2.564102564102564,
+      "grad_norm": 2.5454132556915283,
+      "learning_rate": 1.0424412031961483e-06,
+      "logits/chosen": 0.091796875,
+      "logits/rejected": 0.140625,
+      "logps/chosen": -133.0,
+      "logps/rejected": -159.0,
+      "loss": 0.4611,
+      "rewards/accuracies": 0.8333333134651184,
+      "rewards/chosen": -1.1640625,
+      "rewards/margins": 1.2265625,
+      "rewards/rejected": -2.390625,
+      "step": 300
+    },
+    {
+      "epoch": 2.58974358974359,
+      "grad_norm": 2.596614360809326,
+      "learning_rate": 1.025469618366351e-06,
+      "logits/chosen": 0.10693359375,
+      "logits/rejected": 0.12255859375,
+      "logps/chosen": -127.0,
+      "logps/rejected": -139.0,
+      "loss": 0.3776,
+      "rewards/accuracies": 0.875,
+      "rewards/chosen": -1.046875,
+      "rewards/margins": 1.3125,
+      "rewards/rejected": -2.359375,
+      "step": 303
+    },
+    {
+      "epoch": 2.6153846153846154,
+      "grad_norm": 2.499739170074463,
+      "learning_rate": 1.0084906889341656e-06,
+      "logits/chosen": 0.1259765625,
+      "logits/rejected": 0.126953125,
+      "logps/chosen": -119.5,
+      "logps/rejected": -153.0,
+      "loss": 0.4089,
+      "rewards/accuracies": 0.875,
+      "rewards/chosen": -1.1171875,
+      "rewards/margins": 1.15625,
+      "rewards/rejected": -2.28125,
+      "step": 306
+    },
+    {
+      "epoch": 2.641025641025641,
+      "grad_norm": 2.699960231781006,
+      "learning_rate": 9.915093110658345e-07,
+      "logits/chosen": 0.0986328125,
+      "logits/rejected": 0.08349609375,
+      "logps/chosen": -130.0,
+      "logps/rejected": -157.0,
+      "loss": 0.3958,
+      "rewards/accuracies": 0.8333333134651184,
+      "rewards/chosen": -1.4140625,
+      "rewards/margins": 1.1484375,
+      "rewards/rejected": -2.5625,
+      "step": 309
+    },
+    {
+      "epoch": 2.6666666666666665,
+      "grad_norm": 2.4888288974761963,
+      "learning_rate": 9.745303816336488e-07,
+      "logits/chosen": 0.08447265625,
+      "logits/rejected": 0.1953125,
+      "logps/chosen": -120.5,
+      "logps/rejected": -146.0,
+      "loss": 0.4124,
+      "rewards/accuracies": 0.9583333134651184,
+      "rewards/chosen": -0.875,
+      "rewards/margins": 1.6328125,
+      "rewards/rejected": -2.5,
+      "step": 312
+    },
+    {
+      "epoch": 2.6923076923076925,
+      "grad_norm": 1.9904980659484863,
+      "learning_rate": 9.575587968038518e-07,
+      "logits/chosen": 0.1083984375,
+      "logits/rejected": 0.21484375,
+      "logps/chosen": -125.5,
+      "logps/rejected": -149.0,
+      "loss": 0.3721,
+      "rewards/accuracies": 0.9166666865348816,
+      "rewards/chosen": -0.9453125,
+      "rewards/margins": 1.71875,
+      "rewards/rejected": -2.671875,
+      "step": 315
+    },
+    {
+      "epoch": 2.717948717948718,
+      "grad_norm": 2.642254590988159,
+      "learning_rate": 9.405994506247431e-07,
+      "logits/chosen": 0.1083984375,
+      "logits/rejected": 0.11376953125,
+      "logps/chosen": -138.0,
+      "logps/rejected": -154.0,
+      "loss": 0.3767,
+      "rewards/accuracies": 0.9166666865348816,
+      "rewards/chosen": -1.1171875,
+      "rewards/margins": 1.21875,
+      "rewards/rejected": -2.328125,
+      "step": 318
+    },
+    {
+      "epoch": 2.7435897435897436,
+      "grad_norm": 2.751514434814453,
+      "learning_rate": 9.236572336153985e-07,
+      "logits/chosen": 0.236328125,
+      "logits/rejected": 0.1708984375,
+      "logps/chosen": -108.0,
+      "logps/rejected": -150.0,
+      "loss": 0.3958,
+      "rewards/accuracies": 0.9166666865348816,
+      "rewards/chosen": -0.96484375,
+      "rewards/margins": 1.5,
+      "rewards/rejected": -2.46875,
+      "step": 321
+    },
+    {
+      "epoch": 2.769230769230769,
+      "grad_norm": 2.73346209526062,
+      "learning_rate": 9.067370313554015e-07,
+      "logits/chosen": 0.150390625,
+      "logits/rejected": 0.130859375,
+      "logps/chosen": -107.0,
+      "logps/rejected": -141.0,
+      "loss": 0.3487,
+      "rewards/accuracies": 0.9166666865348816,
+      "rewards/chosen": -0.83203125,
+      "rewards/margins": 1.7109375,
+      "rewards/rejected": -2.546875,
+      "step": 324
+    },
+    {
+      "epoch": 2.7948717948717947,
+      "grad_norm": 2.5458078384399414,
+      "learning_rate": 8.898437230760057e-07,
+      "logits/chosen": 0.208984375,
+      "logits/rejected": 0.13671875,
+      "logps/chosen": -110.5,
+      "logps/rejected": -152.0,
+      "loss": 0.3914,
+      "rewards/accuracies": 0.9166666865348816,
+      "rewards/chosen": -0.89453125,
+      "rewards/margins": 1.40625,
+      "rewards/rejected": -2.296875,
+      "step": 327
+    },
+    {
+      "epoch": 2.8205128205128203,
+      "grad_norm": 2.728673219680786,
+      "learning_rate": 8.729821802531212e-07,
+      "logits/chosen": 0.146484375,
+      "logits/rejected": 0.1123046875,
+      "logps/chosen": -136.0,
+      "logps/rejected": -158.0,
+      "loss": 0.419,
+      "rewards/accuracies": 0.875,
+      "rewards/chosen": -1.15625,
+      "rewards/margins": 1.4453125,
+      "rewards/rejected": -2.59375,
+      "step": 330
+    },
+    {
+      "epoch": 2.8461538461538463,
+      "grad_norm": 2.601808786392212,
+      "learning_rate": 8.561572652025446e-07,
+      "logits/chosen": 0.130859375,
+      "logits/rejected": 0.103515625,
+      "logps/chosen": -117.5,
+      "logps/rejected": -150.0,
+      "loss": 0.372,
+      "rewards/accuracies": 0.8333333134651184,
+      "rewards/chosen": -1.109375,
+      "rewards/margins": 1.3046875,
+      "rewards/rejected": -2.40625,
+      "step": 333
+    },
+    {
+      "epoch": 2.871794871794872,
+      "grad_norm": 2.5382025241851807,
+      "learning_rate": 8.393738296778227e-07,
+      "logits/chosen": 0.04931640625,
+      "logits/rejected": 0.0257568359375,
+      "logps/chosen": -136.0,
+      "logps/rejected": -136.0,
+      "loss": 0.4305,
+      "rewards/accuracies": 0.7083333134651184,
+      "rewards/chosen": -1.265625,
+      "rewards/margins": 0.7265625,
+      "rewards/rejected": -1.9921875,
+      "step": 336
+    },
+    {
+      "epoch": 2.8974358974358974,
+      "grad_norm": 2.9267117977142334,
+      "learning_rate": 8.226367134711691e-07,
+      "logits/chosen": 0.046875,
+      "logits/rejected": 0.12890625,
+      "logps/chosen": -129.0,
+      "logps/rejected": -160.0,
+      "loss": 0.4168,
+      "rewards/accuracies": 0.7916666865348816,
+      "rewards/chosen": -1.3046875,
+      "rewards/margins": 1.46875,
+      "rewards/rejected": -2.765625,
+      "step": 339
+    },
+    {
+      "epoch": 2.9230769230769234,
+      "grad_norm": 2.3900935649871826,
+      "learning_rate": 8.059507430178246e-07,
+      "logits/chosen": 0.080078125,
+      "logits/rejected": 0.1220703125,
+      "logps/chosen": -146.0,
+      "logps/rejected": -175.0,
+      "loss": 0.4027,
+      "rewards/accuracies": 0.7916666865348816,
+      "rewards/chosen": -1.390625,
+      "rewards/margins": 1.609375,
+      "rewards/rejected": -3.0,
+      "step": 342
+    },
+    {
+      "epoch": 2.948717948717949,
+      "grad_norm": 2.6986002922058105,
+      "learning_rate": 7.89320730004274e-07,
+      "logits/chosen": 0.1103515625,
+      "logits/rejected": 0.08056640625,
+      "logps/chosen": -105.0,
+      "logps/rejected": -145.0,
+      "loss": 0.4405,
+      "rewards/accuracies": 0.875,
+      "rewards/chosen": -1.2265625,
+      "rewards/margins": 1.2109375,
+      "rewards/rejected": -2.4375,
+      "step": 345
+    },
+    {
+      "epoch": 2.9743589743589745,
+      "grad_norm": 2.854278564453125,
+      "learning_rate": 7.727514699807101e-07,
+      "logits/chosen": 0.1748046875,
+      "logits/rejected": 0.1796875,
+      "logps/chosen": -116.0,
+      "logps/rejected": -127.0,
+      "loss": 0.394,
+      "rewards/accuracies": 0.7916666865348816,
+      "rewards/chosen": -0.93359375,
+      "rewards/margins": 1.09375,
+      "rewards/rejected": -2.015625,
+      "step": 348
+    },
+    {
+      "epoch": 3.0,
+      "grad_norm": 2.017193078994751,
+      "learning_rate": 7.562477409781584e-07,
+      "logits/chosen": 0.035888671875,
+      "logits/rejected": 0.08935546875,
+      "logps/chosen": -107.0,
+      "logps/rejected": -125.0,
+      "loss": 0.3628,
+      "rewards/accuracies": 0.7916666865348816,
+      "rewards/chosen": -0.92578125,
+      "rewards/margins": 1.109375,
+      "rewards/rejected": -2.03125,
+      "step": 351
+    },
+    {
+      "epoch": 3.0256410256410255,
+      "grad_norm": 2.873739242553711,
+      "learning_rate": 7.39814302130647e-07,
+      "logits/chosen": 0.1572265625,
+      "logits/rejected": 0.1005859375,
+      "logps/chosen": -117.5,
+      "logps/rejected": -140.0,
+      "loss": 0.399,
+      "rewards/accuracies": 0.9166666865348816,
+      "rewards/chosen": -0.890625,
+      "rewards/margins": 1.546875,
+      "rewards/rejected": -2.4375,
+      "step": 354
+    },
+    {
+      "epoch": 3.051282051282051,
+      "grad_norm": 2.0300047397613525,
+      "learning_rate": 7.234558923028289e-07,
+      "logits/chosen": 0.0693359375,
+      "logits/rejected": 0.09228515625,
+      "logps/chosen": -116.0,
+      "logps/rejected": -146.0,
+      "loss": 0.3313,
+      "rewards/accuracies": 0.875,
+      "rewards/chosen": -1.046875,
+      "rewards/margins": 1.5078125,
+      "rewards/rejected": -2.5625,
+      "step": 357
+    },
+    {
+      "epoch": 3.076923076923077,
+      "grad_norm": 3.1289193630218506,
+      "learning_rate": 7.071772287234496e-07,
+      "logits/chosen": 0.001678466796875,
+      "logits/rejected": 0.0296630859375,
+      "logps/chosen": -138.0,
+      "logps/rejected": -174.0,
+      "loss": 0.3334,
+      "rewards/accuracies": 0.9166666865348816,
+      "rewards/chosen": -1.1953125,
+      "rewards/margins": 1.640625,
+      "rewards/rejected": -2.84375,
+      "step": 360
+    },
+    {
+      "epoch": 3.1025641025641026,
+      "grad_norm": 2.7747201919555664,
+      "learning_rate": 6.909830056250526e-07,
+      "logits/chosen": 0.0093994140625,
+      "logits/rejected": 0.05322265625,
+      "logps/chosen": -139.0,
+      "logps/rejected": -159.0,
+      "loss": 0.3814,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": -1.203125,
+      "rewards/margins": 1.46875,
+      "rewards/rejected": -2.6875,
+      "step": 363
+    },
+    {
+      "epoch": 3.128205128205128,
+      "grad_norm": 2.415590286254883,
+      "learning_rate": 6.748778928903165e-07,
+      "logits/chosen": 0.056396484375,
+      "logits/rejected": 0.11181640625,
+      "logps/chosen": -123.0,
+      "logps/rejected": -163.0,
+      "loss": 0.3505,
+      "rewards/accuracies": 0.9583333134651184,
+      "rewards/chosen": -1.2265625,
+      "rewards/margins": 1.8515625,
+      "rewards/rejected": -3.078125,
+      "step": 366
+    },
+    {
+      "epoch": 3.1538461538461537,
+      "grad_norm": 2.7427213191986084,
+      "learning_rate": 6.588665347054152e-07,
+      "logits/chosen": -0.031494140625,
+      "logits/rejected": 0.06494140625,
+      "logps/chosen": -135.0,
+      "logps/rejected": -162.0,
+      "loss": 0.3808,
+      "rewards/accuracies": 0.9166666865348816,
+      "rewards/chosen": -1.21875,
+      "rewards/margins": 1.578125,
+      "rewards/rejected": -2.796875,
+      "step": 369
+    },
+    {
+      "epoch": 3.1794871794871793,
+      "grad_norm": 2.864881753921509,
+      "learning_rate": 6.429535482207846e-07,
+      "logits/chosen": 0.09716796875,
+      "logits/rejected": 0.07958984375,
+      "logps/chosen": -112.0,
+      "logps/rejected": -140.0,
+      "loss": 0.354,
+      "rewards/accuracies": 0.9166666865348816,
+      "rewards/chosen": -0.96875,
+      "rewards/margins": 1.453125,
+      "rewards/rejected": -2.421875,
+      "step": 372
+    },
+    {
+      "epoch": 3.2051282051282053,
+      "grad_norm": 2.073596954345703,
+      "learning_rate": 6.271435222196914e-07,
+      "logits/chosen": 0.0291748046875,
+      "logits/rejected": 0.009033203125,
+      "logps/chosen": -139.0,
+      "logps/rejected": -149.0,
+      "loss": 0.315,
+      "rewards/accuracies": 0.875,
+      "rewards/chosen": -1.2578125,
+      "rewards/margins": 1.6328125,
+      "rewards/rejected": -2.890625,
+      "step": 375
+    },
+    {
+      "epoch": 3.230769230769231,
+      "grad_norm": 2.8894355297088623,
+      "learning_rate": 6.114410157949745e-07,
+      "logits/chosen": 0.018798828125,
+      "logits/rejected": 0.0986328125,
+      "logps/chosen": -123.0,
+      "logps/rejected": -159.0,
+      "loss": 0.4066,
+      "rewards/accuracies": 0.7916666865348816,
+      "rewards/chosen": -1.3125,
+      "rewards/margins": 1.3359375,
+      "rewards/rejected": -2.640625,
+      "step": 378
+    },
+    {
+      "epoch": 3.2564102564102564,
+      "grad_norm": 3.8194832801818848,
+      "learning_rate": 5.958505570343561e-07,
+      "logits/chosen": 0.1689453125,
+      "logits/rejected": 0.1611328125,
+      "logps/chosen": -110.5,
+      "logps/rejected": -136.0,
+      "loss": 0.4066,
+      "rewards/accuracies": 0.75,
+      "rewards/chosen": -1.34375,
+      "rewards/margins": 0.91015625,
+      "rewards/rejected": -2.25,
+      "step": 381
+    },
+    {
+      "epoch": 3.282051282051282,
+      "grad_norm": 2.2550456523895264,
+      "learning_rate": 5.803766417146859e-07,
+      "logits/chosen": 0.1865234375,
+      "logits/rejected": 0.1357421875,
+      "logps/chosen": -114.0,
+      "logps/rejected": -158.0,
+      "loss": 0.3373,
+      "rewards/accuracies": 0.9166666865348816,
+      "rewards/chosen": -1.3046875,
+      "rewards/margins": 1.75,
+      "rewards/rejected": -3.0625,
+      "step": 384
+    },
+    {
+      "epoch": 3.3076923076923075,
+      "grad_norm": 3.2783937454223633,
+      "learning_rate": 5.650237320055106e-07,
+      "logits/chosen": -0.030029296875,
+      "logits/rejected": 0.04052734375,
+      "logps/chosen": -141.0,
+      "logps/rejected": -160.0,
+      "loss": 0.3154,
+      "rewards/accuracies": 0.9583333134651184,
+      "rewards/chosen": -1.3984375,
+      "rewards/margins": 1.796875,
+      "rewards/rejected": -3.203125,
+      "step": 387
+    },
+    {
+      "epoch": 3.3333333333333335,
+      "grad_norm": 3.349325180053711,
+      "learning_rate": 5.497962551823266e-07,
+      "logits/chosen": -0.0537109375,
+      "logits/rejected": 0.0213623046875,
+      "logps/chosen": -140.0,
+      "logps/rejected": -175.0,
+      "loss": 0.3752,
+      "rewards/accuracies": 0.875,
+      "rewards/chosen": -1.359375,
+      "rewards/margins": 1.625,
+      "rewards/rejected": -2.984375,
+      "step": 390
+    },
+    {
+      "epoch": 3.358974358974359,
+      "grad_norm": 2.396205186843872,
+      "learning_rate": 5.346986023499026e-07,
+      "logits/chosen": 0.1240234375,
+      "logits/rejected": 0.126953125,
+      "logps/chosen": -135.0,
+      "logps/rejected": -171.0,
+      "loss": 0.3165,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": -1.328125,
+      "rewards/margins": 1.8984375,
+      "rewards/rejected": -3.21875,
+      "step": 393
+    },
+    {
+      "epoch": 3.3846153846153846,
+      "grad_norm": 2.4508631229400635,
+      "learning_rate": 5.197351271760258e-07,
+      "logits/chosen": -0.0263671875,
+      "logits/rejected": 0.0966796875,
+      "logps/chosen": -116.0,
+      "logps/rejected": -130.0,
+      "loss": 0.3593,
+      "rewards/accuracies": 0.875,
+      "rewards/chosen": -0.9609375,
+      "rewards/margins": 1.7578125,
+      "rewards/rejected": -2.71875,
+      "step": 396
+    },
+    {
+      "epoch": 3.41025641025641,
+      "grad_norm": 3.2673799991607666,
+      "learning_rate": 5.049101446360497e-07,
+      "logits/chosen": 0.134765625,
+      "logits/rejected": 0.2021484375,
+      "logps/chosen": -110.5,
+      "logps/rejected": -133.0,
+      "loss": 0.3686,
+      "rewards/accuracies": 0.875,
+      "rewards/chosen": -0.98828125,
+      "rewards/margins": 1.65625,
+      "rewards/rejected": -2.640625,
+      "step": 399
+    },
+    {
+      "epoch": 3.435897435897436,
+      "grad_norm": 2.6491222381591797,
+      "learning_rate": 4.902279297685945e-07,
+      "logits/chosen": 0.0693359375,
+      "logits/rejected": 0.1455078125,
+      "logps/chosen": -124.0,
+      "logps/rejected": -137.0,
+      "loss": 0.3663,
+      "rewards/accuracies": 0.875,
+      "rewards/chosen": -1.0546875,
+      "rewards/margins": 1.359375,
+      "rewards/rejected": -2.40625,
+      "step": 402
+    },
+    {
+      "epoch": 3.4615384615384617,
+      "grad_norm": 3.2985990047454834,
+      "learning_rate": 4.756927164427684e-07,
+      "logits/chosen": 0.12451171875,
+      "logits/rejected": 0.1474609375,
+      "logps/chosen": -122.0,
+      "logps/rejected": -163.0,
+      "loss": 0.3685,
+      "rewards/accuracies": 0.9166666865348816,
+      "rewards/chosen": -1.3515625,
+      "rewards/margins": 1.4609375,
+      "rewards/rejected": -2.8125,
+      "step": 405
+    },
+    {
+      "epoch": 3.4871794871794872,
+      "grad_norm": 1.697457194328308,
+      "learning_rate": 4.613086961372561e-07,
+      "logits/chosen": 0.181640625,
+      "logits/rejected": 0.23046875,
+      "logps/chosen": -113.0,
+      "logps/rejected": -124.0,
+      "loss": 0.333,
+      "rewards/accuracies": 0.8333333134651184,
+      "rewards/chosen": -1.3828125,
+      "rewards/margins": 1.0703125,
+      "rewards/rejected": -2.453125,
+      "step": 408
+    },
+    {
+      "epoch": 3.5128205128205128,
+      "grad_norm": 3.6975581645965576,
+      "learning_rate": 4.4708001673163663e-07,
+      "logits/chosen": 0.0174560546875,
+      "logits/rejected": 0.04052734375,
+      "logps/chosen": -120.0,
+      "logps/rejected": -150.0,
+      "loss": 0.3828,
+      "rewards/accuracies": 0.9583333134651184,
+      "rewards/chosen": -1.125,
+      "rewards/margins": 1.7890625,
+      "rewards/rejected": -2.921875,
+      "step": 411
+    },
+    {
+      "epoch": 3.5384615384615383,
+      "grad_norm": 2.6693131923675537,
+      "learning_rate": 4.3301078131026823e-07,
+      "logits/chosen": 0.072265625,
+      "logits/rejected": 0.08740234375,
+      "logps/chosen": -140.0,
+      "logps/rejected": -170.0,
+      "loss": 0.3223,
+      "rewards/accuracies": 0.9166666865348816,
+      "rewards/chosen": -1.4765625,
+      "rewards/margins": 1.9140625,
+      "rewards/rejected": -3.390625,
+      "step": 414
+    },
+    {
+      "epoch": 3.564102564102564,
+      "grad_norm": 3.120739221572876,
+      "learning_rate": 4.191050469790961e-07,
+      "logits/chosen": 0.09033203125,
+      "logits/rejected": 0.12353515625,
+      "logps/chosen": -123.0,
+      "logps/rejected": -155.0,
+      "loss": 0.3776,
+      "rewards/accuracies": 0.7916666865348816,
+      "rewards/chosen": -1.5,
+      "rewards/margins": 1.3984375,
+      "rewards/rejected": -2.890625,
+      "step": 417
+    },
+    {
+      "epoch": 3.58974358974359,
+      "grad_norm": 2.7248075008392334,
+      "learning_rate": 4.053668236957134e-07,
+      "logits/chosen": 0.072265625,
+      "logits/rejected": 0.09765625,
+      "logps/chosen": -134.0,
+      "logps/rejected": -165.0,
+      "loss": 0.2954,
+      "rewards/accuracies": 0.9583333134651184,
+      "rewards/chosen": -1.265625,
+      "rewards/margins": 2.1875,
+      "rewards/rejected": -3.453125,
+      "step": 420
+    },
+    {
+      "epoch": 3.6153846153846154,
+      "grad_norm": 2.782973527908325,
+      "learning_rate": 3.918000731130238e-07,
+      "logits/chosen": 0.1015625,
+      "logits/rejected": 0.123046875,
+      "logps/chosen": -115.0,
+      "logps/rejected": -152.0,
+      "loss": 0.336,
+      "rewards/accuracies": 0.8333333134651184,
+      "rewards/chosen": -1.203125,
+      "rewards/margins": 1.828125,
+      "rewards/rejected": -3.03125,
+      "step": 423
+    },
+    {
+      "epoch": 3.641025641025641,
+      "grad_norm": 2.623746395111084,
+      "learning_rate": 3.7840870743682784e-07,
+      "logits/chosen": 0.0361328125,
+      "logits/rejected": -0.0135498046875,
+      "logps/chosen": -114.5,
+      "logps/rejected": -144.0,
+      "loss": 0.332,
+      "rewards/accuracies": 0.8333333134651184,
+      "rewards/chosen": -1.1796875,
+      "rewards/margins": 1.4140625,
+      "rewards/rejected": -2.59375,
+      "step": 426
+    },
+    {
+      "epoch": 3.6666666666666665,
+      "grad_norm": 3.1472394466400146,
+      "learning_rate": 3.651965882976745e-07,
+      "logits/chosen": 0.076171875,
+      "logits/rejected": 0.061279296875,
+      "logps/chosen": -121.0,
+      "logps/rejected": -138.0,
+      "loss": 0.4179,
+      "rewards/accuracies": 0.7083333134651184,
+      "rewards/chosen": -1.5546875,
+      "rewards/margins": 1.1015625,
+      "rewards/rejected": -2.65625,
+      "step": 429
+    },
+    {
+      "epoch": 3.6923076923076925,
+      "grad_norm": 3.3787548542022705,
+      "learning_rate": 3.5216752563729004e-07,
+      "logits/chosen": 0.05126953125,
+      "logits/rejected": 0.09716796875,
+      "logps/chosen": -125.0,
+      "logps/rejected": -144.0,
+      "loss": 0.3857,
+      "rewards/accuracies": 0.8333333134651184,
+      "rewards/chosen": -1.5,
+      "rewards/margins": 1.546875,
+      "rewards/rejected": -3.046875,
+      "step": 432
+    },
+    {
+      "epoch": 3.717948717948718,
+      "grad_norm": 2.2831146717071533,
+      "learning_rate": 3.393252766099187e-07,
+      "logits/chosen": -0.06103515625,
+      "logits/rejected": -0.0380859375,
+      "logps/chosen": -129.0,
+      "logps/rejected": -160.0,
+      "loss": 0.2803,
+      "rewards/accuracies": 0.9583333134651184,
+      "rewards/chosen": -1.2890625,
+      "rewards/margins": 1.984375,
+      "rewards/rejected": -3.265625,
+      "step": 435
+    },
+    {
+      "epoch": 3.7435897435897436,
+      "grad_norm": 3.6543710231781006,
+      "learning_rate": 3.2667354449888074e-07,
+      "logits/chosen": 0.00030517578125,
+      "logits/rejected": 0.039794921875,
+      "logps/chosen": -146.0,
+      "logps/rejected": -165.0,
+      "loss": 0.4561,
+      "rewards/accuracies": 0.875,
+      "rewards/chosen": -1.8984375,
+      "rewards/margins": 1.2109375,
+      "rewards/rejected": -3.109375,
+      "step": 438
+    },
+    {
+      "epoch": 3.769230769230769,
+      "grad_norm": 2.645735025405884,
+      "learning_rate": 3.1421597764866856e-07,
+      "logits/chosen": 0.04248046875,
+      "logits/rejected": 0.0380859375,
+      "logps/chosen": -105.5,
+      "logps/rejected": -143.0,
+      "loss": 0.3253,
+      "rewards/accuracies": 0.875,
+      "rewards/chosen": -1.046875,
+      "rewards/margins": 1.609375,
+      "rewards/rejected": -2.65625,
+      "step": 441
+    },
+    {
+      "epoch": 3.7948717948717947,
+      "grad_norm": 3.269747018814087,
+      "learning_rate": 3.019561684128823e-07,
+      "logits/chosen": 0.040771484375,
+      "logits/rejected": 0.005584716796875,
+      "logps/chosen": -123.5,
+      "logps/rejected": -143.0,
+      "loss": 0.4199,
+      "rewards/accuracies": 0.8333333134651184,
+      "rewards/chosen": -1.25,
+      "rewards/margins": 1.2890625,
+      "rewards/rejected": -2.546875,
+      "step": 444
+    },
+    {
+      "epoch": 3.8205128205128203,
+      "grad_norm": 3.2382609844207764,
+      "learning_rate": 2.898976521183143e-07,
+      "logits/chosen": 0.1064453125,
+      "logits/rejected": 0.0830078125,
+      "logps/chosen": -120.5,
+      "logps/rejected": -149.0,
+      "loss": 0.3449,
+      "rewards/accuracies": 0.8333333134651184,
+      "rewards/chosen": -1.1328125,
+      "rewards/margins": 1.8125,
+      "rewards/rejected": -2.9375,
+      "step": 447
+    },
+    {
+      "epoch": 3.8461538461538463,
+      "grad_norm": 3.1709492206573486,
+      "learning_rate": 2.7804390604547556e-07,
+      "logits/chosen": -0.0198974609375,
+      "logits/rejected": -0.01123046875,
+      "logps/chosen": -135.0,
+      "logps/rejected": -170.0,
+      "loss": 0.3787,
+      "rewards/accuracies": 0.875,
+      "rewards/chosen": -1.65625,
+      "rewards/margins": 2.0,
+      "rewards/rejected": -3.65625,
+      "step": 450
+    },
+    {
+      "epoch": 3.871794871794872,
+      "grad_norm": 2.88985013961792,
+      "learning_rate": 2.663983484258636e-07,
+      "logits/chosen": 0.031494140625,
+      "logits/rejected": 0.048583984375,
+      "logps/chosen": -137.0,
+      "logps/rejected": -157.0,
+      "loss": 0.3285,
+      "rewards/accuracies": 0.9166666865348816,
+      "rewards/chosen": -1.6484375,
+      "rewards/margins": 1.5859375,
+      "rewards/rejected": -3.234375,
+      "step": 453
+    },
+    {
+      "epoch": 3.8974358974358974,
+      "grad_norm": 3.0768532752990723,
+      "learning_rate": 2.549643374562549e-07,
+      "logits/chosen": 0.1064453125,
+      "logits/rejected": 0.1376953125,
+      "logps/chosen": -90.5,
+      "logps/rejected": -125.0,
+      "loss": 0.3256,
+      "rewards/accuracies": 0.875,
+      "rewards/chosen": -1.140625,
+      "rewards/margins": 1.359375,
+      "rewards/rejected": -2.5,
+      "step": 456
+    },
+    {
+      "epoch": 3.9230769230769234,
+      "grad_norm": 2.6071791648864746,
+      "learning_rate": 2.4374517033031283e-07,
+      "logits/chosen": 0.1171875,
+      "logits/rejected": 0.14453125,
+      "logps/chosen": -121.0,
+      "logps/rejected": -146.0,
+      "loss": 0.3638,
+      "rewards/accuracies": 0.8333333134651184,
+      "rewards/chosen": -1.3359375,
+      "rewards/margins": 1.5546875,
+      "rewards/rejected": -2.890625,
+      "step": 459
+    },
+    {
+      "epoch": 3.948717948717949,
+      "grad_norm": 2.748244285583496,
+      "learning_rate": 2.3274408228778354e-07,
+      "logits/chosen": 0.015625,
+      "logits/rejected": 0.042236328125,
+      "logps/chosen": -112.0,
+      "logps/rejected": -135.0,
+      "loss": 0.3545,
+      "rewards/accuracies": 0.8333333134651184,
+      "rewards/chosen": -0.9375,
+      "rewards/margins": 1.390625,
+      "rewards/rejected": -2.328125,
+      "step": 462
+    },
+    {
+      "epoch": 3.9743589743589745,
+      "grad_norm": 2.53035306930542,
+      "learning_rate": 2.219642456815607e-07,
+      "logits/chosen": 0.1640625,
+      "logits/rejected": 0.0693359375,
+      "logps/chosen": -110.0,
+      "logps/rejected": -133.0,
+      "loss": 0.3363,
+      "rewards/accuracies": 0.9166666865348816,
+      "rewards/chosen": -0.95703125,
+      "rewards/margins": 1.6328125,
+      "rewards/rejected": -2.59375,
+      "step": 465
+    },
+    {
+      "epoch": 4.0,
+      "grad_norm": 2.739515542984009,
+      "learning_rate": 2.1140876906288086e-07,
+      "logits/chosen": 0.005340576171875,
+      "logits/rejected": 0.028076171875,
+      "logps/chosen": -119.0,
+      "logps/rejected": -154.0,
+      "loss": 0.285,
+      "rewards/accuracies": 0.875,
+      "rewards/chosen": -1.3203125,
+      "rewards/margins": 1.6484375,
+      "rewards/rejected": -2.96875,
+      "step": 468
+    }
+  ],
+  "logging_steps": 3,
+  "max_steps": 585,
+  "num_input_tokens_seen": 0,
+  "num_train_epochs": 5,
+  "save_steps": 500,
+  "stateful_callbacks": {
+    "TrainerControl": {
+      "args": {
+        "should_epoch_stop": false,
+        "should_evaluate": false,
+        "should_log": false,
+        "should_save": true,
+        "should_training_stop": false
+      },
+      "attributes": {}
+    }
+  },
+  "total_flos": 0.0,
+  "train_batch_size": 4,
+  "trial_name": null,
+  "trial_params": null
+}

7b/iteration2/training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:492227d3d4411d4317ac7f67aad6b04d8ed2a75d50ca234d1e1b1bb7c6bd3317
+size 7992