Upload 71 files

second_run_dpo/model/README.md

@@ -0,0 +1,202 @@
+---
+base_model: mistralai/Mistral-7B-Instruct-v0.3
+library_name: peft
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.12.1.dev0

second_run_dpo/model/adapter_config.json

@@ -0,0 +1,31 @@
+{
+  "alpha_pattern": {},
+  "auto_mapping": null,
+  "base_model_name_or_path": "mistralai/Mistral-7B-Instruct-v0.3",
+  "bias": "none",
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 16,
+  "lora_dropout": 0.1,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "r": 64,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "q_proj",
+    "v_proj",
+    "k_proj",
+    "o_proj"
+  ],
+  "task_type": "CAUSAL_LM",
+  "use_dora": false,
+  "use_rslora": false
+}

second_run_dpo/model/adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:92d63dd46155ee09134e2a478e57a91e155358c5d4e3326e72224e062dbb3db7
+size 218138576

second_run_dpo/model/checkpoint-100/README.md

@@ -0,0 +1,202 @@
+---
+base_model: mistralai/Mistral-7B-Instruct-v0.3
+library_name: peft
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.12.1.dev0

second_run_dpo/model/checkpoint-100/adapter_config.json

@@ -0,0 +1,31 @@
+{
+  "alpha_pattern": {},
+  "auto_mapping": null,
+  "base_model_name_or_path": "mistralai/Mistral-7B-Instruct-v0.3",
+  "bias": "none",
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 16,
+  "lora_dropout": 0.1,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "r": 64,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "q_proj",
+    "v_proj",
+    "k_proj",
+    "o_proj"
+  ],
+  "task_type": "CAUSAL_LM",
+  "use_dora": false,
+  "use_rslora": false
+}

second_run_dpo/model/checkpoint-100/adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d19a7649675dde556f1d8a0c066fefb9b77dc64be309d0001177c5172d34f585
+size 218138576

second_run_dpo/model/checkpoint-100/optimizer.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e9c24b5e0899a744157c133d934df84245c23884d3340d45f028a7668027e564
+size 1511285882

second_run_dpo/model/checkpoint-100/rng_state.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:386fcc8cc1089aade9450d86fb239ea3483f455fd2d78d8378645feecfec9d69
+size 14244

second_run_dpo/model/checkpoint-100/scheduler.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:329c12cb5a53e925e5e134cc6bcabcf38425643325df07365f1c6f110adf931b
+size 1064

second_run_dpo/model/checkpoint-100/special_tokens_map.json

@@ -0,0 +1,24 @@
+{
+  "bos_token": {
+    "content": "<s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "</s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": "</s>",
+  "unk_token": {
+    "content": "<unk>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

second_run_dpo/model/checkpoint-100/tokenizer.model

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:37f00374dea48658ee8f5d0f21895b9bc55cb0103939607c8185bfd1c6ca1f89
+size 587404

second_run_dpo/model/checkpoint-100/trainer_state.json

@@ -0,0 +1,183 @@
+{
+  "best_metric": null,
+  "best_model_checkpoint": null,
+  "epoch": 0.4214963119072708,
+  "eval_steps": 500,
+  "global_step": 100,
+  "is_hyper_param_search": false,
+  "is_local_process_zero": true,
+  "is_world_process_zero": true,
+  "log_history": [
+    {
+      "epoch": 0.04214963119072708,
+      "grad_norm": 54.265682220458984,
+      "learning_rate": 4.789029535864979e-06,
+      "logits/chosen": -3.230823516845703,
+      "logits/rejected": -3.1391842365264893,
+      "logps/chosen": -155.92660522460938,
+      "logps/rejected": -108.29136657714844,
+      "loss": 0.8901,
+      "rewards/accuracies": 0.625,
+      "rewards/chosen": 0.2371383160352707,
+      "rewards/margins": 0.4253893494606018,
+      "rewards/rejected": -0.18825098872184753,
+      "step": 10
+    },
+    {
+      "epoch": 0.08429926238145416,
+      "grad_norm": 0.18127574026584625,
+      "learning_rate": 4.578059071729958e-06,
+      "logits/chosen": -3.2088217735290527,
+      "logits/rejected": -3.1587116718292236,
+      "logps/chosen": -133.78977966308594,
+      "logps/rejected": -120.8590087890625,
+      "loss": 0.0638,
+      "rewards/accuracies": 0.925000011920929,
+      "rewards/chosen": 3.8257803916931152,
+      "rewards/margins": 5.4808030128479,
+      "rewards/rejected": -1.655022382736206,
+      "step": 20
+    },
+    {
+      "epoch": 0.12644889357218125,
+      "grad_norm": 0.008206675760447979,
+      "learning_rate": 4.367088607594937e-06,
+      "logits/chosen": -3.157487392425537,
+      "logits/rejected": -3.1080830097198486,
+      "logps/chosen": -137.19386291503906,
+      "logps/rejected": -148.9159393310547,
+      "loss": 0.0348,
+      "rewards/accuracies": 0.949999988079071,
+      "rewards/chosen": 4.677396774291992,
+      "rewards/margins": 10.149096488952637,
+      "rewards/rejected": -5.4716997146606445,
+      "step": 30
+    },
+    {
+      "epoch": 0.16859852476290832,
+      "grad_norm": 25.140769958496094,
+      "learning_rate": 4.156118143459915e-06,
+      "logits/chosen": -3.146048069000244,
+      "logits/rejected": -3.095085382461548,
+      "logps/chosen": -126.72493743896484,
+      "logps/rejected": -149.604248046875,
+      "loss": 0.0575,
+      "rewards/accuracies": 0.925000011920929,
+      "rewards/chosen": 5.172249794006348,
+      "rewards/margins": 11.624964714050293,
+      "rewards/rejected": -6.452713966369629,
+      "step": 40
+    },
+    {
+      "epoch": 0.2107481559536354,
+      "grad_norm": 0.00036443868884816766,
+      "learning_rate": 3.945147679324895e-06,
+      "logits/chosen": -3.125558853149414,
+      "logits/rejected": -3.073667526245117,
+      "logps/chosen": -126.54759216308594,
+      "logps/rejected": -154.06100463867188,
+      "loss": 0.0607,
+      "rewards/accuracies": 0.9125000238418579,
+      "rewards/chosen": 5.867059230804443,
+      "rewards/margins": 12.704333305358887,
+      "rewards/rejected": -6.837274074554443,
+      "step": 50
+    },
+    {
+      "epoch": 0.2528977871443625,
+      "grad_norm": 0.0066557377576828,
+      "learning_rate": 3.7341772151898737e-06,
+      "logits/chosen": -3.1266560554504395,
+      "logits/rejected": -3.0419058799743652,
+      "logps/chosen": -123.27961730957031,
+      "logps/rejected": -133.74404907226562,
+      "loss": 0.0762,
+      "rewards/accuracies": 0.8999999761581421,
+      "rewards/chosen": 5.405282497406006,
+      "rewards/margins": 9.596057891845703,
+      "rewards/rejected": -4.190774440765381,
+      "step": 60
+    },
+    {
+      "epoch": 0.2950474183350896,
+      "grad_norm": 5.82294225692749,
+      "learning_rate": 3.523206751054853e-06,
+      "logits/chosen": -3.134911298751831,
+      "logits/rejected": -3.0802435874938965,
+      "logps/chosen": -114.3982162475586,
+      "logps/rejected": -137.26548767089844,
+      "loss": 0.0676,
+      "rewards/accuracies": 0.9125000238418579,
+      "rewards/chosen": 5.872519493103027,
+      "rewards/margins": 10.642888069152832,
+      "rewards/rejected": -4.770368576049805,
+      "step": 70
+    },
+    {
+      "epoch": 0.33719704952581664,
+      "grad_norm": 0.005566390231251717,
+      "learning_rate": 3.3122362869198316e-06,
+      "logits/chosen": -3.118222713470459,
+      "logits/rejected": -3.0644006729125977,
+      "logps/chosen": -110.6181411743164,
+      "logps/rejected": -131.0067901611328,
+      "loss": 0.0867,
+      "rewards/accuracies": 0.875,
+      "rewards/chosen": 6.600110054016113,
+      "rewards/margins": 10.749144554138184,
+      "rewards/rejected": -4.149035453796387,
+      "step": 80
+    },
+    {
+      "epoch": 0.3793466807165437,
+      "grad_norm": 0.0022323213052004576,
+      "learning_rate": 3.10126582278481e-06,
+      "logits/chosen": -3.1222400665283203,
+      "logits/rejected": -3.0632846355438232,
+      "logps/chosen": -114.68511962890625,
+      "logps/rejected": -136.66403198242188,
+      "loss": 0.0522,
+      "rewards/accuracies": 0.925000011920929,
+      "rewards/chosen": 7.240682125091553,
+      "rewards/margins": 11.513819694519043,
+      "rewards/rejected": -4.273137092590332,
+      "step": 90
+    },
+    {
+      "epoch": 0.4214963119072708,
+      "grad_norm": 0.0042571211233735085,
+      "learning_rate": 2.8902953586497895e-06,
+      "logits/chosen": -3.0972230434417725,
+      "logits/rejected": -3.047482490539551,
+      "logps/chosen": -104.4890365600586,
+      "logps/rejected": -127.5343246459961,
+      "loss": 0.104,
+      "rewards/accuracies": 0.8500000238418579,
+      "rewards/chosen": 7.045135498046875,
+      "rewards/margins": 11.117444038391113,
+      "rewards/rejected": -4.07230806350708,
+      "step": 100
+    }
+  ],
+  "logging_steps": 10,
+  "max_steps": 237,
+  "num_input_tokens_seen": 0,
+  "num_train_epochs": 1,
+  "save_steps": 50,
+  "stateful_callbacks": {
+    "TrainerControl": {
+      "args": {
+        "should_epoch_stop": false,
+        "should_evaluate": false,
+        "should_log": false,
+        "should_save": true,
+        "should_training_stop": false
+      },
+      "attributes": {}
+    }
+  },
+  "total_flos": 0.0,
+  "train_batch_size": 2,
+  "trial_name": null,
+  "trial_params": null
+}

second_run_dpo/model/checkpoint-100/training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:30cd87144cef67ea1dc1be92b1a808040175102b9708a50175edca8393637ce2
+size 5944

second_run_dpo/model/checkpoint-150/README.md

@@ -0,0 +1,202 @@
+---
+base_model: mistralai/Mistral-7B-Instruct-v0.3
+library_name: peft
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.12.1.dev0

second_run_dpo/model/checkpoint-150/adapter_config.json

@@ -0,0 +1,31 @@
+{
+  "alpha_pattern": {},
+  "auto_mapping": null,
+  "base_model_name_or_path": "mistralai/Mistral-7B-Instruct-v0.3",
+  "bias": "none",
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 16,
+  "lora_dropout": 0.1,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "r": 64,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "q_proj",
+    "v_proj",
+    "k_proj",
+    "o_proj"
+  ],
+  "task_type": "CAUSAL_LM",
+  "use_dora": false,
+  "use_rslora": false
+}

second_run_dpo/model/checkpoint-150/adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3b259fe592eef289641cc95ae6a96a350eccdd14e7cede0320e38376afec9659
+size 218138576

second_run_dpo/model/checkpoint-150/optimizer.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d692788e7dc22e1e825548e7c26a949802fb8f13c0e03f6442bccb6cad20e4eb
+size 1511285882

second_run_dpo/model/checkpoint-150/rng_state.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:386fcc8cc1089aade9450d86fb239ea3483f455fd2d78d8378645feecfec9d69
+size 14244

second_run_dpo/model/checkpoint-150/scheduler.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ec764fee333a791ccb3c2fa4e2889206cb1a013c0403a368d2f7a9344f06a0d2
+size 1064

second_run_dpo/model/checkpoint-150/special_tokens_map.json

@@ -0,0 +1,24 @@
+{
+  "bos_token": {
+    "content": "<s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "</s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": "</s>",
+  "unk_token": {
+    "content": "<unk>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

second_run_dpo/model/checkpoint-150/tokenizer.model

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:37f00374dea48658ee8f5d0f21895b9bc55cb0103939607c8185bfd1c6ca1f89
+size 587404

second_run_dpo/model/checkpoint-150/trainer_state.json

@@ -0,0 +1,258 @@
+{
+  "best_metric": null,
+  "best_model_checkpoint": null,
+  "epoch": 0.6322444678609063,
+  "eval_steps": 500,
+  "global_step": 150,
+  "is_hyper_param_search": false,
+  "is_local_process_zero": true,
+  "is_world_process_zero": true,
+  "log_history": [
+    {
+      "epoch": 0.04214963119072708,
+      "grad_norm": 54.265682220458984,
+      "learning_rate": 4.789029535864979e-06,
+      "logits/chosen": -3.230823516845703,
+      "logits/rejected": -3.1391842365264893,
+      "logps/chosen": -155.92660522460938,
+      "logps/rejected": -108.29136657714844,
+      "loss": 0.8901,
+      "rewards/accuracies": 0.625,
+      "rewards/chosen": 0.2371383160352707,
+      "rewards/margins": 0.4253893494606018,
+      "rewards/rejected": -0.18825098872184753,
+      "step": 10
+    },
+    {
+      "epoch": 0.08429926238145416,
+      "grad_norm": 0.18127574026584625,
+      "learning_rate": 4.578059071729958e-06,
+      "logits/chosen": -3.2088217735290527,
+      "logits/rejected": -3.1587116718292236,
+      "logps/chosen": -133.78977966308594,
+      "logps/rejected": -120.8590087890625,
+      "loss": 0.0638,
+      "rewards/accuracies": 0.925000011920929,
+      "rewards/chosen": 3.8257803916931152,
+      "rewards/margins": 5.4808030128479,
+      "rewards/rejected": -1.655022382736206,
+      "step": 20
+    },
+    {
+      "epoch": 0.12644889357218125,
+      "grad_norm": 0.008206675760447979,
+      "learning_rate": 4.367088607594937e-06,
+      "logits/chosen": -3.157487392425537,
+      "logits/rejected": -3.1080830097198486,
+      "logps/chosen": -137.19386291503906,
+      "logps/rejected": -148.9159393310547,
+      "loss": 0.0348,
+      "rewards/accuracies": 0.949999988079071,
+      "rewards/chosen": 4.677396774291992,
+      "rewards/margins": 10.149096488952637,
+      "rewards/rejected": -5.4716997146606445,
+      "step": 30
+    },
+    {
+      "epoch": 0.16859852476290832,
+      "grad_norm": 25.140769958496094,
+      "learning_rate": 4.156118143459915e-06,
+      "logits/chosen": -3.146048069000244,
+      "logits/rejected": -3.095085382461548,
+      "logps/chosen": -126.72493743896484,
+      "logps/rejected": -149.604248046875,
+      "loss": 0.0575,
+      "rewards/accuracies": 0.925000011920929,
+      "rewards/chosen": 5.172249794006348,
+      "rewards/margins": 11.624964714050293,
+      "rewards/rejected": -6.452713966369629,
+      "step": 40
+    },
+    {
+      "epoch": 0.2107481559536354,
+      "grad_norm": 0.00036443868884816766,
+      "learning_rate": 3.945147679324895e-06,
+      "logits/chosen": -3.125558853149414,
+      "logits/rejected": -3.073667526245117,
+      "logps/chosen": -126.54759216308594,
+      "logps/rejected": -154.06100463867188,
+      "loss": 0.0607,
+      "rewards/accuracies": 0.9125000238418579,
+      "rewards/chosen": 5.867059230804443,
+      "rewards/margins": 12.704333305358887,
+      "rewards/rejected": -6.837274074554443,
+      "step": 50
+    },
+    {
+      "epoch": 0.2528977871443625,
+      "grad_norm": 0.0066557377576828,
+      "learning_rate": 3.7341772151898737e-06,
+      "logits/chosen": -3.1266560554504395,
+      "logits/rejected": -3.0419058799743652,
+      "logps/chosen": -123.27961730957031,
+      "logps/rejected": -133.74404907226562,
+      "loss": 0.0762,
+      "rewards/accuracies": 0.8999999761581421,
+      "rewards/chosen": 5.405282497406006,
+      "rewards/margins": 9.596057891845703,
+      "rewards/rejected": -4.190774440765381,
+      "step": 60
+    },
+    {
+      "epoch": 0.2950474183350896,
+      "grad_norm": 5.82294225692749,
+      "learning_rate": 3.523206751054853e-06,
+      "logits/chosen": -3.134911298751831,
+      "logits/rejected": -3.0802435874938965,
+      "logps/chosen": -114.3982162475586,
+      "logps/rejected": -137.26548767089844,
+      "loss": 0.0676,
+      "rewards/accuracies": 0.9125000238418579,
+      "rewards/chosen": 5.872519493103027,
+      "rewards/margins": 10.642888069152832,
+      "rewards/rejected": -4.770368576049805,
+      "step": 70
+    },
+    {
+      "epoch": 0.33719704952581664,
+      "grad_norm": 0.005566390231251717,
+      "learning_rate": 3.3122362869198316e-06,
+      "logits/chosen": -3.118222713470459,
+      "logits/rejected": -3.0644006729125977,
+      "logps/chosen": -110.6181411743164,
+      "logps/rejected": -131.0067901611328,
+      "loss": 0.0867,
+      "rewards/accuracies": 0.875,
+      "rewards/chosen": 6.600110054016113,
+      "rewards/margins": 10.749144554138184,
+      "rewards/rejected": -4.149035453796387,
+      "step": 80
+    },
+    {
+      "epoch": 0.3793466807165437,
+      "grad_norm": 0.0022323213052004576,
+      "learning_rate": 3.10126582278481e-06,
+      "logits/chosen": -3.1222400665283203,
+      "logits/rejected": -3.0632846355438232,
+      "logps/chosen": -114.68511962890625,
+      "logps/rejected": -136.66403198242188,
+      "loss": 0.0522,
+      "rewards/accuracies": 0.925000011920929,
+      "rewards/chosen": 7.240682125091553,
+      "rewards/margins": 11.513819694519043,
+      "rewards/rejected": -4.273137092590332,
+      "step": 90
+    },
+    {
+      "epoch": 0.4214963119072708,
+      "grad_norm": 0.0042571211233735085,
+      "learning_rate": 2.8902953586497895e-06,
+      "logits/chosen": -3.0972230434417725,
+      "logits/rejected": -3.047482490539551,
+      "logps/chosen": -104.4890365600586,
+      "logps/rejected": -127.5343246459961,
+      "loss": 0.104,
+      "rewards/accuracies": 0.8500000238418579,
+      "rewards/chosen": 7.045135498046875,
+      "rewards/margins": 11.117444038391113,
+      "rewards/rejected": -4.07230806350708,
+      "step": 100
+    },
+    {
+      "epoch": 0.4636459430979979,
+      "grad_norm": 0.010770019143819809,
+      "learning_rate": 2.679324894514768e-06,
+      "logits/chosen": -3.084117889404297,
+      "logits/rejected": -3.0263478755950928,
+      "logps/chosen": -109.60150146484375,
+      "logps/rejected": -138.71580505371094,
+      "loss": 0.0461,
+      "rewards/accuracies": 0.9375,
+      "rewards/chosen": 7.922713279724121,
+      "rewards/margins": 12.225544929504395,
+      "rewards/rejected": -4.302830696105957,
+      "step": 110
+    },
+    {
+      "epoch": 0.505795574288725,
+      "grad_norm": 0.03155478835105896,
+      "learning_rate": 2.4683544303797473e-06,
+      "logits/chosen": -3.0575947761535645,
+      "logits/rejected": -3.00960111618042,
+      "logps/chosen": -98.39952850341797,
+      "logps/rejected": -125.97001647949219,
+      "loss": 0.0968,
+      "rewards/accuracies": 0.862500011920929,
+      "rewards/chosen": 7.3856916427612305,
+      "rewards/margins": 11.279303550720215,
+      "rewards/rejected": -3.8936104774475098,
+      "step": 120
+    },
+    {
+      "epoch": 0.547945205479452,
+      "grad_norm": 0.0017593610100448132,
+      "learning_rate": 2.257383966244726e-06,
+      "logits/chosen": -3.0633864402770996,
+      "logits/rejected": -3.003227710723877,
+      "logps/chosen": -112.51448822021484,
+      "logps/rejected": -144.1097412109375,
+      "loss": 0.026,
+      "rewards/accuracies": 0.9624999761581421,
+      "rewards/chosen": 8.68992805480957,
+      "rewards/margins": 13.341931343078613,
+      "rewards/rejected": -4.652005195617676,
+      "step": 130
+    },
+    {
+      "epoch": 0.5900948366701791,
+      "grad_norm": 0.032057471573352814,
+      "learning_rate": 2.046413502109705e-06,
+      "logits/chosen": -3.0402822494506836,
+      "logits/rejected": -2.9848685264587402,
+      "logps/chosen": -109.23091888427734,
+      "logps/rejected": -137.82015991210938,
+      "loss": 0.0521,
+      "rewards/accuracies": 0.925000011920929,
+      "rewards/chosen": 8.288289070129395,
+      "rewards/margins": 12.732525825500488,
+      "rewards/rejected": -4.44423770904541,
+      "step": 140
+    },
+    {
+      "epoch": 0.6322444678609063,
+      "grad_norm": 0.0025352409575134516,
+      "learning_rate": 1.8354430379746838e-06,
+      "logits/chosen": -3.0412521362304688,
+      "logits/rejected": -2.9849700927734375,
+      "logps/chosen": -103.49588775634766,
+      "logps/rejected": -131.9301300048828,
+      "loss": 0.0781,
+      "rewards/accuracies": 0.887499988079071,
+      "rewards/chosen": 8.078404426574707,
+      "rewards/margins": 12.365747451782227,
+      "rewards/rejected": -4.287342071533203,
+      "step": 150
+    }
+  ],
+  "logging_steps": 10,
+  "max_steps": 237,
+  "num_input_tokens_seen": 0,
+  "num_train_epochs": 1,
+  "save_steps": 50,
+  "stateful_callbacks": {
+    "TrainerControl": {
+      "args": {
+        "should_epoch_stop": false,
+        "should_evaluate": false,
+        "should_log": false,
+        "should_save": true,
+        "should_training_stop": false
+      },
+      "attributes": {}
+    }
+  },
+  "total_flos": 0.0,
+  "train_batch_size": 2,
+  "trial_name": null,
+  "trial_params": null
+}

second_run_dpo/model/checkpoint-150/training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:30cd87144cef67ea1dc1be92b1a808040175102b9708a50175edca8393637ce2
+size 5944

second_run_dpo/model/checkpoint-200/README.md

@@ -0,0 +1,202 @@
+---
+base_model: mistralai/Mistral-7B-Instruct-v0.3
+library_name: peft
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.12.1.dev0

second_run_dpo/model/checkpoint-200/adapter_config.json

@@ -0,0 +1,31 @@
+{
+  "alpha_pattern": {},
+  "auto_mapping": null,
+  "base_model_name_or_path": "mistralai/Mistral-7B-Instruct-v0.3",
+  "bias": "none",
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 16,
+  "lora_dropout": 0.1,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "r": 64,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "q_proj",
+    "v_proj",
+    "k_proj",
+    "o_proj"
+  ],
+  "task_type": "CAUSAL_LM",
+  "use_dora": false,
+  "use_rslora": false
+}

second_run_dpo/model/checkpoint-200/adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c1eb2458736060d2897a06b602df5ded34b870fd6b2b9bb56e46c3dd01174617
+size 218138576

second_run_dpo/model/checkpoint-200/optimizer.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ac308bbb0b85906a2d3553d8569ce5a0499b9afb8bfce4e2738aa1cec8f34f49
+size 1511285882

second_run_dpo/model/checkpoint-200/rng_state.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:386fcc8cc1089aade9450d86fb239ea3483f455fd2d78d8378645feecfec9d69
+size 14244

second_run_dpo/model/checkpoint-200/scheduler.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d1cd8e9027f2241499093d395717a6007aad3b8ae633e6b192fd1c031d3edc30
+size 1064

second_run_dpo/model/checkpoint-200/special_tokens_map.json

@@ -0,0 +1,24 @@
+{
+  "bos_token": {
+    "content": "<s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "</s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": "</s>",
+  "unk_token": {
+    "content": "<unk>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

second_run_dpo/model/checkpoint-200/tokenizer.model

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:37f00374dea48658ee8f5d0f21895b9bc55cb0103939607c8185bfd1c6ca1f89
+size 587404

second_run_dpo/model/checkpoint-200/trainer_state.json

@@ -0,0 +1,333 @@
+{
+  "best_metric": null,
+  "best_model_checkpoint": null,
+  "epoch": 0.8429926238145417,
+  "eval_steps": 500,
+  "global_step": 200,
+  "is_hyper_param_search": false,
+  "is_local_process_zero": true,
+  "is_world_process_zero": true,
+  "log_history": [
+    {
+      "epoch": 0.04214963119072708,
+      "grad_norm": 54.265682220458984,
+      "learning_rate": 4.789029535864979e-06,
+      "logits/chosen": -3.230823516845703,
+      "logits/rejected": -3.1391842365264893,
+      "logps/chosen": -155.92660522460938,
+      "logps/rejected": -108.29136657714844,
+      "loss": 0.8901,
+      "rewards/accuracies": 0.625,
+      "rewards/chosen": 0.2371383160352707,
+      "rewards/margins": 0.4253893494606018,
+      "rewards/rejected": -0.18825098872184753,
+      "step": 10
+    },
+    {
+      "epoch": 0.08429926238145416,
+      "grad_norm": 0.18127574026584625,
+      "learning_rate": 4.578059071729958e-06,
+      "logits/chosen": -3.2088217735290527,
+      "logits/rejected": -3.1587116718292236,
+      "logps/chosen": -133.78977966308594,
+      "logps/rejected": -120.8590087890625,
+      "loss": 0.0638,
+      "rewards/accuracies": 0.925000011920929,
+      "rewards/chosen": 3.8257803916931152,
+      "rewards/margins": 5.4808030128479,
+      "rewards/rejected": -1.655022382736206,
+      "step": 20
+    },
+    {
+      "epoch": 0.12644889357218125,
+      "grad_norm": 0.008206675760447979,
+      "learning_rate": 4.367088607594937e-06,
+      "logits/chosen": -3.157487392425537,
+      "logits/rejected": -3.1080830097198486,
+      "logps/chosen": -137.19386291503906,
+      "logps/rejected": -148.9159393310547,
+      "loss": 0.0348,
+      "rewards/accuracies": 0.949999988079071,
+      "rewards/chosen": 4.677396774291992,
+      "rewards/margins": 10.149096488952637,
+      "rewards/rejected": -5.4716997146606445,
+      "step": 30
+    },
+    {
+      "epoch": 0.16859852476290832,
+      "grad_norm": 25.140769958496094,
+      "learning_rate": 4.156118143459915e-06,
+      "logits/chosen": -3.146048069000244,
+      "logits/rejected": -3.095085382461548,
+      "logps/chosen": -126.72493743896484,
+      "logps/rejected": -149.604248046875,
+      "loss": 0.0575,
+      "rewards/accuracies": 0.925000011920929,
+      "rewards/chosen": 5.172249794006348,
+      "rewards/margins": 11.624964714050293,
+      "rewards/rejected": -6.452713966369629,
+      "step": 40
+    },
+    {
+      "epoch": 0.2107481559536354,
+      "grad_norm": 0.00036443868884816766,
+      "learning_rate": 3.945147679324895e-06,
+      "logits/chosen": -3.125558853149414,
+      "logits/rejected": -3.073667526245117,
+      "logps/chosen": -126.54759216308594,
+      "logps/rejected": -154.06100463867188,
+      "loss": 0.0607,
+      "rewards/accuracies": 0.9125000238418579,
+      "rewards/chosen": 5.867059230804443,
+      "rewards/margins": 12.704333305358887,
+      "rewards/rejected": -6.837274074554443,
+      "step": 50
+    },
+    {
+      "epoch": 0.2528977871443625,
+      "grad_norm": 0.0066557377576828,
+      "learning_rate": 3.7341772151898737e-06,
+      "logits/chosen": -3.1266560554504395,
+      "logits/rejected": -3.0419058799743652,
+      "logps/chosen": -123.27961730957031,
+      "logps/rejected": -133.74404907226562,
+      "loss": 0.0762,
+      "rewards/accuracies": 0.8999999761581421,
+      "rewards/chosen": 5.405282497406006,
+      "rewards/margins": 9.596057891845703,
+      "rewards/rejected": -4.190774440765381,
+      "step": 60
+    },
+    {
+      "epoch": 0.2950474183350896,
+      "grad_norm": 5.82294225692749,
+      "learning_rate": 3.523206751054853e-06,
+      "logits/chosen": -3.134911298751831,
+      "logits/rejected": -3.0802435874938965,
+      "logps/chosen": -114.3982162475586,
+      "logps/rejected": -137.26548767089844,
+      "loss": 0.0676,
+      "rewards/accuracies": 0.9125000238418579,
+      "rewards/chosen": 5.872519493103027,
+      "rewards/margins": 10.642888069152832,
+      "rewards/rejected": -4.770368576049805,
+      "step": 70
+    },
+    {
+      "epoch": 0.33719704952581664,
+      "grad_norm": 0.005566390231251717,
+      "learning_rate": 3.3122362869198316e-06,
+      "logits/chosen": -3.118222713470459,
+      "logits/rejected": -3.0644006729125977,
+      "logps/chosen": -110.6181411743164,
+      "logps/rejected": -131.0067901611328,
+      "loss": 0.0867,
+      "rewards/accuracies": 0.875,
+      "rewards/chosen": 6.600110054016113,
+      "rewards/margins": 10.749144554138184,
+      "rewards/rejected": -4.149035453796387,
+      "step": 80
+    },
+    {
+      "epoch": 0.3793466807165437,
+      "grad_norm": 0.0022323213052004576,
+      "learning_rate": 3.10126582278481e-06,
+      "logits/chosen": -3.1222400665283203,
+      "logits/rejected": -3.0632846355438232,
+      "logps/chosen": -114.68511962890625,
+      "logps/rejected": -136.66403198242188,
+      "loss": 0.0522,
+      "rewards/accuracies": 0.925000011920929,
+      "rewards/chosen": 7.240682125091553,
+      "rewards/margins": 11.513819694519043,
+      "rewards/rejected": -4.273137092590332,
+      "step": 90
+    },
+    {
+      "epoch": 0.4214963119072708,
+      "grad_norm": 0.0042571211233735085,
+      "learning_rate": 2.8902953586497895e-06,
+      "logits/chosen": -3.0972230434417725,
+      "logits/rejected": -3.047482490539551,
+      "logps/chosen": -104.4890365600586,
+      "logps/rejected": -127.5343246459961,
+      "loss": 0.104,
+      "rewards/accuracies": 0.8500000238418579,
+      "rewards/chosen": 7.045135498046875,
+      "rewards/margins": 11.117444038391113,
+      "rewards/rejected": -4.07230806350708,
+      "step": 100
+    },
+    {
+      "epoch": 0.4636459430979979,
+      "grad_norm": 0.010770019143819809,
+      "learning_rate": 2.679324894514768e-06,
+      "logits/chosen": -3.084117889404297,
+      "logits/rejected": -3.0263478755950928,
+      "logps/chosen": -109.60150146484375,
+      "logps/rejected": -138.71580505371094,
+      "loss": 0.0461,
+      "rewards/accuracies": 0.9375,
+      "rewards/chosen": 7.922713279724121,
+      "rewards/margins": 12.225544929504395,
+      "rewards/rejected": -4.302830696105957,
+      "step": 110
+    },
+    {
+      "epoch": 0.505795574288725,
+      "grad_norm": 0.03155478835105896,
+      "learning_rate": 2.4683544303797473e-06,
+      "logits/chosen": -3.0575947761535645,
+      "logits/rejected": -3.00960111618042,
+      "logps/chosen": -98.39952850341797,
+      "logps/rejected": -125.97001647949219,
+      "loss": 0.0968,
+      "rewards/accuracies": 0.862500011920929,
+      "rewards/chosen": 7.3856916427612305,
+      "rewards/margins": 11.279303550720215,
+      "rewards/rejected": -3.8936104774475098,
+      "step": 120
+    },
+    {
+      "epoch": 0.547945205479452,
+      "grad_norm": 0.0017593610100448132,
+      "learning_rate": 2.257383966244726e-06,
+      "logits/chosen": -3.0633864402770996,
+      "logits/rejected": -3.003227710723877,
+      "logps/chosen": -112.51448822021484,
+      "logps/rejected": -144.1097412109375,
+      "loss": 0.026,
+      "rewards/accuracies": 0.9624999761581421,
+      "rewards/chosen": 8.68992805480957,
+      "rewards/margins": 13.341931343078613,
+      "rewards/rejected": -4.652005195617676,
+      "step": 130
+    },
+    {
+      "epoch": 0.5900948366701791,
+      "grad_norm": 0.032057471573352814,
+      "learning_rate": 2.046413502109705e-06,
+      "logits/chosen": -3.0402822494506836,
+      "logits/rejected": -2.9848685264587402,
+      "logps/chosen": -109.23091888427734,
+      "logps/rejected": -137.82015991210938,
+      "loss": 0.0521,
+      "rewards/accuracies": 0.925000011920929,
+      "rewards/chosen": 8.288289070129395,
+      "rewards/margins": 12.732525825500488,
+      "rewards/rejected": -4.44423770904541,
+      "step": 140
+    },
+    {
+      "epoch": 0.6322444678609063,
+      "grad_norm": 0.0025352409575134516,
+      "learning_rate": 1.8354430379746838e-06,
+      "logits/chosen": -3.0412521362304688,
+      "logits/rejected": -2.9849700927734375,
+      "logps/chosen": -103.49588775634766,
+      "logps/rejected": -131.9301300048828,
+      "loss": 0.0781,
+      "rewards/accuracies": 0.887499988079071,
+      "rewards/chosen": 8.078404426574707,
+      "rewards/margins": 12.365747451782227,
+      "rewards/rejected": -4.287342071533203,
+      "step": 150
+    },
+    {
+      "epoch": 0.6743940990516333,
+      "grad_norm": 0.0029362805653363466,
+      "learning_rate": 1.6244725738396625e-06,
+      "logits/chosen": -3.030087947845459,
+      "logits/rejected": -2.977440595626831,
+      "logps/chosen": -101.0843734741211,
+      "logps/rejected": -132.79702758789062,
+      "loss": 0.0783,
+      "rewards/accuracies": 0.887499988079071,
+      "rewards/chosen": 7.972678184509277,
+      "rewards/margins": 12.395901679992676,
+      "rewards/rejected": -4.42322301864624,
+      "step": 160
+    },
+    {
+      "epoch": 0.7165437302423604,
+      "grad_norm": 0.30753812193870544,
+      "learning_rate": 1.4135021097046414e-06,
+      "logits/chosen": -3.030521869659424,
+      "logits/rejected": -2.9768686294555664,
+      "logps/chosen": -102.2951889038086,
+      "logps/rejected": -133.8944549560547,
+      "loss": 0.056,
+      "rewards/accuracies": 0.925000011920929,
+      "rewards/chosen": 8.018686294555664,
+      "rewards/margins": 12.17985725402832,
+      "rewards/rejected": -4.1611714363098145,
+      "step": 170
+    },
+    {
+      "epoch": 0.7586933614330874,
+      "grad_norm": 0.0034459459129720926,
+      "learning_rate": 1.2025316455696204e-06,
+      "logits/chosen": -3.019465923309326,
+      "logits/rejected": -2.9687447547912598,
+      "logps/chosen": -107.70234680175781,
+      "logps/rejected": -138.5956573486328,
+      "loss": 0.0434,
+      "rewards/accuracies": 0.9375,
+      "rewards/chosen": 8.414751052856445,
+      "rewards/margins": 12.751376152038574,
+      "rewards/rejected": -4.336625576019287,
+      "step": 180
+    },
+    {
+      "epoch": 0.8008429926238145,
+      "grad_norm": 0.001862135250121355,
+      "learning_rate": 9.915611814345991e-07,
+      "logits/chosen": -3.006066083908081,
+      "logits/rejected": -2.943457841873169,
+      "logps/chosen": -109.46881103515625,
+      "logps/rejected": -139.15029907226562,
+      "loss": 0.0433,
+      "rewards/accuracies": 0.9375,
+      "rewards/chosen": 8.667418479919434,
+      "rewards/margins": 12.866633415222168,
+      "rewards/rejected": -4.199214458465576,
+      "step": 190
+    },
+    {
+      "epoch": 0.8429926238145417,
+      "grad_norm": 0.001758416648954153,
+      "learning_rate": 7.805907172995782e-07,
+      "logits/chosen": -2.995535373687744,
+      "logits/rejected": -2.943047285079956,
+      "logps/chosen": -98.70401763916016,
+      "logps/rejected": -130.56216430664062,
+      "loss": 0.0866,
+      "rewards/accuracies": 0.875,
+      "rewards/chosen": 8.245943069458008,
+      "rewards/margins": 12.186595916748047,
+      "rewards/rejected": -3.9406521320343018,
+      "step": 200
+    }
+  ],
+  "logging_steps": 10,
+  "max_steps": 237,
+  "num_input_tokens_seen": 0,
+  "num_train_epochs": 1,
+  "save_steps": 50,
+  "stateful_callbacks": {
+    "TrainerControl": {
+      "args": {
+        "should_epoch_stop": false,
+        "should_evaluate": false,
+        "should_log": false,
+        "should_save": true,
+        "should_training_stop": false
+      },
+      "attributes": {}
+    }
+  },
+  "total_flos": 0.0,
+  "train_batch_size": 2,
+  "trial_name": null,
+  "trial_params": null
+}

second_run_dpo/model/checkpoint-200/training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:30cd87144cef67ea1dc1be92b1a808040175102b9708a50175edca8393637ce2
+size 5944

second_run_dpo/model/checkpoint-237/README.md

@@ -0,0 +1,202 @@
+---
+base_model: mistralai/Mistral-7B-Instruct-v0.3
+library_name: peft
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.12.1.dev0

second_run_dpo/model/checkpoint-237/adapter_config.json

@@ -0,0 +1,31 @@
+{
+  "alpha_pattern": {},
+  "auto_mapping": null,
+  "base_model_name_or_path": "mistralai/Mistral-7B-Instruct-v0.3",
+  "bias": "none",
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 16,
+  "lora_dropout": 0.1,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "r": 64,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "q_proj",
+    "v_proj",
+    "k_proj",
+    "o_proj"
+  ],
+  "task_type": "CAUSAL_LM",
+  "use_dora": false,
+  "use_rslora": false
+}

second_run_dpo/model/checkpoint-237/adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:92d63dd46155ee09134e2a478e57a91e155358c5d4e3326e72224e062dbb3db7
+size 218138576

second_run_dpo/model/checkpoint-237/optimizer.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:acda6ad6d9c0a5a311e160ef07e634d2c3fc720806f6036587fbd162877b6c7d
+size 1511285882

second_run_dpo/model/checkpoint-237/rng_state.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:386fcc8cc1089aade9450d86fb239ea3483f455fd2d78d8378645feecfec9d69
+size 14244

second_run_dpo/model/checkpoint-237/scheduler.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3ade0a6cc3b8a8c2abce94feb4e1d48f24e163fd69b55fcc47a1ec5ac50e438f
+size 1064

second_run_dpo/model/checkpoint-237/special_tokens_map.json

@@ -0,0 +1,24 @@
+{
+  "bos_token": {
+    "content": "<s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "</s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": "</s>",
+  "unk_token": {
+    "content": "<unk>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

second_run_dpo/model/checkpoint-237/tokenizer.model

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:37f00374dea48658ee8f5d0f21895b9bc55cb0103939607c8185bfd1c6ca1f89
+size 587404

second_run_dpo/model/checkpoint-237/trainer_state.json

@@ -0,0 +1,378 @@
+{
+  "best_metric": null,
+  "best_model_checkpoint": null,
+  "epoch": 0.9989462592202318,
+  "eval_steps": 500,
+  "global_step": 237,
+  "is_hyper_param_search": false,
+  "is_local_process_zero": true,
+  "is_world_process_zero": true,
+  "log_history": [
+    {
+      "epoch": 0.04214963119072708,
+      "grad_norm": 54.265682220458984,
+      "learning_rate": 4.789029535864979e-06,
+      "logits/chosen": -3.230823516845703,
+      "logits/rejected": -3.1391842365264893,
+      "logps/chosen": -155.92660522460938,
+      "logps/rejected": -108.29136657714844,
+      "loss": 0.8901,
+      "rewards/accuracies": 0.625,
+      "rewards/chosen": 0.2371383160352707,
+      "rewards/margins": 0.4253893494606018,
+      "rewards/rejected": -0.18825098872184753,
+      "step": 10
+    },
+    {
+      "epoch": 0.08429926238145416,
+      "grad_norm": 0.18127574026584625,
+      "learning_rate": 4.578059071729958e-06,
+      "logits/chosen": -3.2088217735290527,
+      "logits/rejected": -3.1587116718292236,
+      "logps/chosen": -133.78977966308594,
+      "logps/rejected": -120.8590087890625,
+      "loss": 0.0638,
+      "rewards/accuracies": 0.925000011920929,
+      "rewards/chosen": 3.8257803916931152,
+      "rewards/margins": 5.4808030128479,
+      "rewards/rejected": -1.655022382736206,
+      "step": 20
+    },
+    {
+      "epoch": 0.12644889357218125,
+      "grad_norm": 0.008206675760447979,
+      "learning_rate": 4.367088607594937e-06,
+      "logits/chosen": -3.157487392425537,
+      "logits/rejected": -3.1080830097198486,
+      "logps/chosen": -137.19386291503906,
+      "logps/rejected": -148.9159393310547,
+      "loss": 0.0348,
+      "rewards/accuracies": 0.949999988079071,
+      "rewards/chosen": 4.677396774291992,
+      "rewards/margins": 10.149096488952637,
+      "rewards/rejected": -5.4716997146606445,
+      "step": 30
+    },
+    {
+      "epoch": 0.16859852476290832,
+      "grad_norm": 25.140769958496094,
+      "learning_rate": 4.156118143459915e-06,
+      "logits/chosen": -3.146048069000244,
+      "logits/rejected": -3.095085382461548,
+      "logps/chosen": -126.72493743896484,
+      "logps/rejected": -149.604248046875,
+      "loss": 0.0575,
+      "rewards/accuracies": 0.925000011920929,
+      "rewards/chosen": 5.172249794006348,
+      "rewards/margins": 11.624964714050293,
+      "rewards/rejected": -6.452713966369629,
+      "step": 40
+    },
+    {
+      "epoch": 0.2107481559536354,
+      "grad_norm": 0.00036443868884816766,
+      "learning_rate": 3.945147679324895e-06,
+      "logits/chosen": -3.125558853149414,
+      "logits/rejected": -3.073667526245117,
+      "logps/chosen": -126.54759216308594,
+      "logps/rejected": -154.06100463867188,
+      "loss": 0.0607,
+      "rewards/accuracies": 0.9125000238418579,
+      "rewards/chosen": 5.867059230804443,
+      "rewards/margins": 12.704333305358887,
+      "rewards/rejected": -6.837274074554443,
+      "step": 50
+    },
+    {
+      "epoch": 0.2528977871443625,
+      "grad_norm": 0.0066557377576828,
+      "learning_rate": 3.7341772151898737e-06,
+      "logits/chosen": -3.1266560554504395,
+      "logits/rejected": -3.0419058799743652,
+      "logps/chosen": -123.27961730957031,
+      "logps/rejected": -133.74404907226562,
+      "loss": 0.0762,
+      "rewards/accuracies": 0.8999999761581421,
+      "rewards/chosen": 5.405282497406006,
+      "rewards/margins": 9.596057891845703,
+      "rewards/rejected": -4.190774440765381,
+      "step": 60
+    },
+    {
+      "epoch": 0.2950474183350896,
+      "grad_norm": 5.82294225692749,
+      "learning_rate": 3.523206751054853e-06,
+      "logits/chosen": -3.134911298751831,
+      "logits/rejected": -3.0802435874938965,
+      "logps/chosen": -114.3982162475586,
+      "logps/rejected": -137.26548767089844,
+      "loss": 0.0676,
+      "rewards/accuracies": 0.9125000238418579,
+      "rewards/chosen": 5.872519493103027,
+      "rewards/margins": 10.642888069152832,
+      "rewards/rejected": -4.770368576049805,
+      "step": 70
+    },
+    {
+      "epoch": 0.33719704952581664,
+      "grad_norm": 0.005566390231251717,
+      "learning_rate": 3.3122362869198316e-06,
+      "logits/chosen": -3.118222713470459,
+      "logits/rejected": -3.0644006729125977,
+      "logps/chosen": -110.6181411743164,
+      "logps/rejected": -131.0067901611328,
+      "loss": 0.0867,
+      "rewards/accuracies": 0.875,
+      "rewards/chosen": 6.600110054016113,
+      "rewards/margins": 10.749144554138184,
+      "rewards/rejected": -4.149035453796387,
+      "step": 80
+    },
+    {
+      "epoch": 0.3793466807165437,
+      "grad_norm": 0.0022323213052004576,
+      "learning_rate": 3.10126582278481e-06,
+      "logits/chosen": -3.1222400665283203,
+      "logits/rejected": -3.0632846355438232,
+      "logps/chosen": -114.68511962890625,
+      "logps/rejected": -136.66403198242188,
+      "loss": 0.0522,
+      "rewards/accuracies": 0.925000011920929,
+      "rewards/chosen": 7.240682125091553,
+      "rewards/margins": 11.513819694519043,
+      "rewards/rejected": -4.273137092590332,
+      "step": 90
+    },
+    {
+      "epoch": 0.4214963119072708,
+      "grad_norm": 0.0042571211233735085,
+      "learning_rate": 2.8902953586497895e-06,
+      "logits/chosen": -3.0972230434417725,
+      "logits/rejected": -3.047482490539551,
+      "logps/chosen": -104.4890365600586,
+      "logps/rejected": -127.5343246459961,
+      "loss": 0.104,
+      "rewards/accuracies": 0.8500000238418579,
+      "rewards/chosen": 7.045135498046875,
+      "rewards/margins": 11.117444038391113,
+      "rewards/rejected": -4.07230806350708,
+      "step": 100
+    },
+    {
+      "epoch": 0.4636459430979979,
+      "grad_norm": 0.010770019143819809,
+      "learning_rate": 2.679324894514768e-06,
+      "logits/chosen": -3.084117889404297,
+      "logits/rejected": -3.0263478755950928,
+      "logps/chosen": -109.60150146484375,
+      "logps/rejected": -138.71580505371094,
+      "loss": 0.0461,
+      "rewards/accuracies": 0.9375,
+      "rewards/chosen": 7.922713279724121,
+      "rewards/margins": 12.225544929504395,
+      "rewards/rejected": -4.302830696105957,
+      "step": 110
+    },
+    {
+      "epoch": 0.505795574288725,
+      "grad_norm": 0.03155478835105896,
+      "learning_rate": 2.4683544303797473e-06,
+      "logits/chosen": -3.0575947761535645,
+      "logits/rejected": -3.00960111618042,
+      "logps/chosen": -98.39952850341797,
+      "logps/rejected": -125.97001647949219,
+      "loss": 0.0968,
+      "rewards/accuracies": 0.862500011920929,
+      "rewards/chosen": 7.3856916427612305,
+      "rewards/margins": 11.279303550720215,
+      "rewards/rejected": -3.8936104774475098,
+      "step": 120
+    },
+    {
+      "epoch": 0.547945205479452,
+      "grad_norm": 0.0017593610100448132,
+      "learning_rate": 2.257383966244726e-06,
+      "logits/chosen": -3.0633864402770996,
+      "logits/rejected": -3.003227710723877,
+      "logps/chosen": -112.51448822021484,
+      "logps/rejected": -144.1097412109375,
+      "loss": 0.026,
+      "rewards/accuracies": 0.9624999761581421,
+      "rewards/chosen": 8.68992805480957,
+      "rewards/margins": 13.341931343078613,
+      "rewards/rejected": -4.652005195617676,
+      "step": 130
+    },
+    {
+      "epoch": 0.5900948366701791,
+      "grad_norm": 0.032057471573352814,
+      "learning_rate": 2.046413502109705e-06,
+      "logits/chosen": -3.0402822494506836,
+      "logits/rejected": -2.9848685264587402,
+      "logps/chosen": -109.23091888427734,
+      "logps/rejected": -137.82015991210938,
+      "loss": 0.0521,
+      "rewards/accuracies": 0.925000011920929,
+      "rewards/chosen": 8.288289070129395,
+      "rewards/margins": 12.732525825500488,
+      "rewards/rejected": -4.44423770904541,
+      "step": 140
+    },
+    {
+      "epoch": 0.6322444678609063,
+      "grad_norm": 0.0025352409575134516,
+      "learning_rate": 1.8354430379746838e-06,
+      "logits/chosen": -3.0412521362304688,
+      "logits/rejected": -2.9849700927734375,
+      "logps/chosen": -103.49588775634766,
+      "logps/rejected": -131.9301300048828,
+      "loss": 0.0781,
+      "rewards/accuracies": 0.887499988079071,
+      "rewards/chosen": 8.078404426574707,
+      "rewards/margins": 12.365747451782227,
+      "rewards/rejected": -4.287342071533203,
+      "step": 150
+    },
+    {
+      "epoch": 0.6743940990516333,
+      "grad_norm": 0.0029362805653363466,
+      "learning_rate": 1.6244725738396625e-06,
+      "logits/chosen": -3.030087947845459,
+      "logits/rejected": -2.977440595626831,
+      "logps/chosen": -101.0843734741211,
+      "logps/rejected": -132.79702758789062,
+      "loss": 0.0783,
+      "rewards/accuracies": 0.887499988079071,
+      "rewards/chosen": 7.972678184509277,
+      "rewards/margins": 12.395901679992676,
+      "rewards/rejected": -4.42322301864624,
+      "step": 160
+    },
+    {
+      "epoch": 0.7165437302423604,
+      "grad_norm": 0.30753812193870544,
+      "learning_rate": 1.4135021097046414e-06,
+      "logits/chosen": -3.030521869659424,
+      "logits/rejected": -2.9768686294555664,
+      "logps/chosen": -102.2951889038086,
+      "logps/rejected": -133.8944549560547,
+      "loss": 0.056,
+      "rewards/accuracies": 0.925000011920929,
+      "rewards/chosen": 8.018686294555664,
+      "rewards/margins": 12.17985725402832,
+      "rewards/rejected": -4.1611714363098145,
+      "step": 170
+    },
+    {
+      "epoch": 0.7586933614330874,
+      "grad_norm": 0.0034459459129720926,
+      "learning_rate": 1.2025316455696204e-06,
+      "logits/chosen": -3.019465923309326,
+      "logits/rejected": -2.9687447547912598,
+      "logps/chosen": -107.70234680175781,
+      "logps/rejected": -138.5956573486328,
+      "loss": 0.0434,
+      "rewards/accuracies": 0.9375,
+      "rewards/chosen": 8.414751052856445,
+      "rewards/margins": 12.751376152038574,
+      "rewards/rejected": -4.336625576019287,
+      "step": 180
+    },
+    {
+      "epoch": 0.8008429926238145,
+      "grad_norm": 0.001862135250121355,
+      "learning_rate": 9.915611814345991e-07,
+      "logits/chosen": -3.006066083908081,
+      "logits/rejected": -2.943457841873169,
+      "logps/chosen": -109.46881103515625,
+      "logps/rejected": -139.15029907226562,
+      "loss": 0.0433,
+      "rewards/accuracies": 0.9375,
+      "rewards/chosen": 8.667418479919434,
+      "rewards/margins": 12.866633415222168,
+      "rewards/rejected": -4.199214458465576,
+      "step": 190
+    },
+    {
+      "epoch": 0.8429926238145417,
+      "grad_norm": 0.001758416648954153,
+      "learning_rate": 7.805907172995782e-07,
+      "logits/chosen": -2.995535373687744,
+      "logits/rejected": -2.943047285079956,
+      "logps/chosen": -98.70401763916016,
+      "logps/rejected": -130.56216430664062,
+      "loss": 0.0866,
+      "rewards/accuracies": 0.875,
+      "rewards/chosen": 8.245943069458008,
+      "rewards/margins": 12.186595916748047,
+      "rewards/rejected": -3.9406521320343018,
+      "step": 200
+    },
+    {
+      "epoch": 0.8851422550052687,
+      "grad_norm": 0.016159718856215477,
+      "learning_rate": 5.69620253164557e-07,
+      "logits/chosen": -3.003826856613159,
+      "logits/rejected": -2.944359302520752,
+      "logps/chosen": -114.41169738769531,
+      "logps/rejected": -147.23350524902344,
+      "loss": 0.0087,
+      "rewards/accuracies": 0.987500011920929,
+      "rewards/chosen": 9.045473098754883,
+      "rewards/margins": 13.605456352233887,
+      "rewards/rejected": -4.559983253479004,
+      "step": 210
+    },
+    {
+      "epoch": 0.9272918861959958,
+      "grad_norm": 0.0007629674510098994,
+      "learning_rate": 3.586497890295359e-07,
+      "logits/chosen": -3.0114877223968506,
+      "logits/rejected": -2.960176467895508,
+      "logps/chosen": -99.81551361083984,
+      "logps/rejected": -132.0089111328125,
+      "loss": 0.0694,
+      "rewards/accuracies": 0.8999999761581421,
+      "rewards/chosen": 8.368062973022461,
+      "rewards/margins": 12.364015579223633,
+      "rewards/rejected": -3.995953321456909,
+      "step": 220
+    },
+    {
+      "epoch": 0.9694415173867229,
+      "grad_norm": 0.5134130716323853,
+      "learning_rate": 1.4767932489451477e-07,
+      "logits/chosen": -3.0056204795837402,
+      "logits/rejected": -2.9448115825653076,
+      "logps/chosen": -107.8029556274414,
+      "logps/rejected": -140.51168823242188,
+      "loss": 0.0267,
+      "rewards/accuracies": 0.9624999761581421,
+      "rewards/chosen": 8.642345428466797,
+      "rewards/margins": 13.103657722473145,
+      "rewards/rejected": -4.461312294006348,
+      "step": 230
+    }
+  ],
+  "logging_steps": 10,
+  "max_steps": 237,
+  "num_input_tokens_seen": 0,
+  "num_train_epochs": 1,
+  "save_steps": 50,
+  "stateful_callbacks": {
+    "TrainerControl": {
+      "args": {
+        "should_epoch_stop": false,
+        "should_evaluate": false,
+        "should_log": false,
+        "should_save": true,
+        "should_training_stop": true
+      },
+      "attributes": {}
+    }
+  },
+  "total_flos": 0.0,
+  "train_batch_size": 2,
+  "trial_name": null,
+  "trial_params": null
+}