Upload folder using huggingface_hub

README.md

@@ -0,0 +1,62 @@
+---
+base_model: codellama/CodeLlama-7b-hf
+library_name: peft
+model_name: exploitgpt-v1
+tags:
+- base_model:adapter:codellama/CodeLlama-7b-hf
+- lora
+- sft
+- transformers
+- trl
+licence: license
+pipeline_tag: text-generation
+---
+
+# Model Card for exploitgpt-v1
+
+This model is a fine-tuned version of [codellama/CodeLlama-7b-hf](https://huggingface.co/codellama/CodeLlama-7b-hf).
+It has been trained using [TRL](https://github.com/huggingface/trl).
+
+## Quick start
+
+```python
+from transformers import pipeline
+
+question = "If you had a time machine, but could only go to the past or the future once and never return, which would you choose and why?"
+generator = pipeline("text-generation", model="None", device="cuda")
+output = generator([{"role": "user", "content": question}], max_new_tokens=128, return_full_text=False)[0]
+print(output["generated_text"])
+```
+
+## Training procedure
+
+ 
+
+
+This model was trained with SFT.
+
+### Framework versions
+
+- PEFT 0.18.0
+- TRL: 0.26.2
+- Transformers: 4.57.3
+- Pytorch: 2.9.1
+- Datasets: 4.4.2
+- Tokenizers: 0.22.1
+
+## Citations
+
+
+
+Cite TRL as:
+    
+```bibtex
+@misc{vonwerra2022trl,
+	title        = {{TRL: Transformer Reinforcement Learning}},
+	author       = {Leandro von Werra and Younes Belkada and Lewis Tunstall and Edward Beeching and Tristan Thrush and Nathan Lambert and Shengyi Huang and Kashif Rasul and Quentin Gallou{\'e}dec},
+	year         = 2020,
+	journal      = {GitHub repository},
+	publisher    = {GitHub},
+	howpublished = {\url{https://github.com/huggingface/trl}}
+}
+```

adapter_config.json

@@ -0,0 +1,41 @@
+{
+  "alora_invocation_tokens": null,
+  "alpha_pattern": {},
+  "arrow_config": null,
+  "auto_mapping": null,
+  "base_model_name_or_path": "codellama/CodeLlama-7b-hf",
+  "bias": "none",
+  "corda_config": null,
+  "ensure_weight_tying": false,
+  "eva_config": null,
+  "exclude_modules": null,
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 32,
+  "lora_bias": false,
+  "lora_dropout": 0.05,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "peft_version": "0.18.0",
+  "qalora_group_size": 16,
+  "r": 16,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "v_proj",
+    "q_proj"
+  ],
+  "target_parameters": null,
+  "task_type": "CAUSAL_LM",
+  "trainable_token_indices": null,
+  "use_dora": false,
+  "use_qalora": false,
+  "use_rslora": false
+}

adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b236c0861c66f1737120aa0df751625931da353a096346b837f509488fb25027
+size 16794456

checkpoint-100/README.md

@@ -0,0 +1,209 @@
+---
+base_model: codellama/CodeLlama-7b-hf
+library_name: peft
+pipeline_tag: text-generation
+tags:
+- base_model:adapter:codellama/CodeLlama-7b-hf
+- lora
+- sft
+- transformers
+- trl
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.18.0

checkpoint-100/adapter_config.json

@@ -0,0 +1,41 @@
+{
+  "alora_invocation_tokens": null,
+  "alpha_pattern": {},
+  "arrow_config": null,
+  "auto_mapping": null,
+  "base_model_name_or_path": "codellama/CodeLlama-7b-hf",
+  "bias": "none",
+  "corda_config": null,
+  "ensure_weight_tying": false,
+  "eva_config": null,
+  "exclude_modules": null,
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 32,
+  "lora_bias": false,
+  "lora_dropout": 0.05,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "peft_version": "0.18.0",
+  "qalora_group_size": 16,
+  "r": 16,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "v_proj",
+    "q_proj"
+  ],
+  "target_parameters": null,
+  "task_type": "CAUSAL_LM",
+  "trainable_token_indices": null,
+  "use_dora": false,
+  "use_qalora": false,
+  "use_rslora": false
+}

checkpoint-100/adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b2c8742fb10f0346fc6b7200f17e64c80cb7c6ae7cf9179c2e6909a22fd62f89
+size 16794456

checkpoint-100/optimizer.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:39b81f6df9216c305843bda8b6967460bd25d22392628c8971798afa73748766
+size 17192011

checkpoint-100/rng_state.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:557bfd295517e7b892bcf30ebd084826730dfcce36462fe02a7439894ed2b8d7
+size 14645

checkpoint-100/scheduler.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:622fb96132554597b206807adb71d93bf8fdf963fa98d12f162324c02badad18
+size 1465

checkpoint-100/special_tokens_map.json

@@ -0,0 +1,30 @@
+{
+  "additional_special_tokens": [
+    "▁<PRE>",
+    "▁<MID>",
+    "▁<SUF>",
+    "▁<EOT>"
+  ],
+  "bos_token": {
+    "content": "<s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "</s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": "</s>",
+  "unk_token": {
+    "content": "<unk>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

checkpoint-100/tokenizer_config.json

@@ -0,0 +1,84 @@
+{
+  "add_bos_token": true,
+  "add_eos_token": false,
+  "added_tokens_decoder": {
+    "0": {
+      "content": "<unk>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "1": {
+      "content": "<s>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "2": {
+      "content": "</s>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "32007": {
+      "content": "▁<PRE>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "32008": {
+      "content": "▁<SUF>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "32009": {
+      "content": "▁<MID>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "32010": {
+      "content": "▁<EOT>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "additional_special_tokens": [
+    "▁<PRE>",
+    "▁<MID>",
+    "▁<SUF>",
+    "▁<EOT>"
+  ],
+  "bos_token": "<s>",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "</s>",
+  "eot_token": "▁<EOT>",
+  "extra_special_tokens": {},
+  "fill_token": "<FILL_ME>",
+  "legacy": null,
+  "middle_token": "▁<MID>",
+  "model_max_length": 1000000000000000019884624838656,
+  "pad_token": "</s>",
+  "prefix_token": "▁<PRE>",
+  "sp_model_kwargs": {},
+  "suffix_token": "▁<SUF>",
+  "tokenizer_class": "CodeLlamaTokenizer",
+  "unk_token": "<unk>",
+  "use_default_system_prompt": false
+}

checkpoint-100/trainer_state.json

@@ -0,0 +1,145 @@
+{
+  "best_global_step": null,
+  "best_metric": null,
+  "best_model_checkpoint": null,
+  "epoch": 0.33984706881903143,
+  "eval_steps": 100,
+  "global_step": 100,
+  "is_hyper_param_search": false,
+  "is_local_process_zero": true,
+  "is_world_process_zero": true,
+  "log_history": [
+    {
+      "entropy": 1.1900819569826127,
+      "epoch": 0.033984706881903144,
+      "grad_norm": 0.107421875,
+      "learning_rate": 6.666666666666667e-05,
+      "loss": 1.2022,
+      "mean_token_accuracy": 0.7261606067419052,
+      "num_tokens": 32329.0,
+      "step": 10
+    },
+    {
+      "entropy": 1.1123934835195541,
+      "epoch": 0.06796941376380629,
+      "grad_norm": 0.138671875,
+      "learning_rate": 0.00014074074074074076,
+      "loss": 1.1277,
+      "mean_token_accuracy": 0.7318252056837082,
+      "num_tokens": 67111.0,
+      "step": 20
+    },
+    {
+      "entropy": 1.0879137217998505,
+      "epoch": 0.10195412064570943,
+      "grad_norm": 0.3203125,
+      "learning_rate": 0.00019999731865174213,
+      "loss": 1.0625,
+      "mean_token_accuracy": 0.7478782564401627,
+      "num_tokens": 101352.0,
+      "step": 30
+    },
+    {
+      "entropy": 0.9986169628798962,
+      "epoch": 0.13593882752761258,
+      "grad_norm": 0.2890625,
+      "learning_rate": 0.00019990348656007261,
+      "loss": 1.0079,
+      "mean_token_accuracy": 0.7581020414829254,
+      "num_tokens": 133883.0,
+      "step": 40
+    },
+    {
+      "entropy": 1.05185257345438,
+      "epoch": 0.16992353440951571,
+      "grad_norm": 0.294921875,
+      "learning_rate": 0.00019967573081342103,
+      "loss": 1.0335,
+      "mean_token_accuracy": 0.7543211042881012,
+      "num_tokens": 168562.0,
+      "step": 50
+    },
+    {
+      "entropy": 0.9918051429092885,
+      "epoch": 0.20390824129141885,
+      "grad_norm": 0.263671875,
+      "learning_rate": 0.00019931435672527624,
+      "loss": 0.9977,
+      "mean_token_accuracy": 0.755571472644806,
+      "num_tokens": 202299.0,
+      "step": 60
+    },
+    {
+      "entropy": 0.8909615643322468,
+      "epoch": 0.23789294817332202,
+      "grad_norm": 0.298828125,
+      "learning_rate": 0.00019881984872856817,
+      "loss": 0.9168,
+      "mean_token_accuracy": 0.7724857151508331,
+      "num_tokens": 237132.0,
+      "step": 70
+    },
+    {
+      "entropy": 0.9534411959350109,
+      "epoch": 0.27187765505522515,
+      "grad_norm": 0.3203125,
+      "learning_rate": 0.00019819286972627066,
+      "loss": 0.9673,
+      "mean_token_accuracy": 0.7637621074914932,
+      "num_tokens": 271644.0,
+      "step": 80
+    },
+    {
+      "entropy": 0.9248364262282849,
+      "epoch": 0.3058623619371283,
+      "grad_norm": 0.26171875,
+      "learning_rate": 0.00019743426020275994,
+      "loss": 0.9077,
+      "mean_token_accuracy": 0.7676862999796867,
+      "num_tokens": 306939.0,
+      "step": 90
+    },
+    {
+      "entropy": 1.0025891564786433,
+      "epoch": 0.33984706881903143,
+      "grad_norm": 0.28125,
+      "learning_rate": 0.00019654503709711982,
+      "loss": 1.0295,
+      "mean_token_accuracy": 0.7536625176668167,
+      "num_tokens": 341079.0,
+      "step": 100
+    },
+    {
+      "epoch": 0.33984706881903143,
+      "eval_entropy": 0.9100993389175052,
+      "eval_loss": 0.9260265231132507,
+      "eval_mean_token_accuracy": 0.7760482584538103,
+      "eval_num_tokens": 341079.0,
+      "eval_runtime": 140.9825,
+      "eval_samples_per_second": 1.043,
+      "eval_steps_per_second": 1.043,
+      "step": 100
+    }
+  ],
+  "logging_steps": 10,
+  "max_steps": 885,
+  "num_input_tokens_seen": 0,
+  "num_train_epochs": 3,
+  "save_steps": 100,
+  "stateful_callbacks": {
+    "TrainerControl": {
+      "args": {
+        "should_epoch_stop": false,
+        "should_evaluate": false,
+        "should_log": false,
+        "should_save": true,
+        "should_training_stop": false
+      },
+      "attributes": {}
+    }
+  },
+  "total_flos": 1.353905810509824e+16,
+  "train_batch_size": 1,
+  "trial_name": null,
+  "trial_params": null
+}

checkpoint-100/training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:be523aaac59d69ae5ec9c8576a8ed2c5b5e19240aee6056559023b48d8fa85f8
+size 6161

checkpoint-200/README.md

@@ -0,0 +1,209 @@
+---
+base_model: codellama/CodeLlama-7b-hf
+library_name: peft
+pipeline_tag: text-generation
+tags:
+- base_model:adapter:codellama/CodeLlama-7b-hf
+- lora
+- sft
+- transformers
+- trl
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.18.0

checkpoint-200/adapter_config.json

@@ -0,0 +1,41 @@
+{
+  "alora_invocation_tokens": null,
+  "alpha_pattern": {},
+  "arrow_config": null,
+  "auto_mapping": null,
+  "base_model_name_or_path": "codellama/CodeLlama-7b-hf",
+  "bias": "none",
+  "corda_config": null,
+  "ensure_weight_tying": false,
+  "eva_config": null,
+  "exclude_modules": null,
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 32,
+  "lora_bias": false,
+  "lora_dropout": 0.05,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "peft_version": "0.18.0",
+  "qalora_group_size": 16,
+  "r": 16,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "v_proj",
+    "q_proj"
+  ],
+  "target_parameters": null,
+  "task_type": "CAUSAL_LM",
+  "trainable_token_indices": null,
+  "use_dora": false,
+  "use_qalora": false,
+  "use_rslora": false
+}

checkpoint-200/adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0338d2c3314786a031bf41c6a629c434537f59af11301d4bae6f1fac6f436f69
+size 16794456

checkpoint-200/optimizer.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:09e85ba91754e7ef62c1417bd60ee10c53cbe28b5bc769745860cf2d81adae35
+size 17192011

checkpoint-200/rng_state.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c236502b3e3509d6ee0c04745ffe5ee03a22bc2da512f7fc7234ad406ddbfecd
+size 14645

checkpoint-200/scheduler.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:cf488b961f4be425dd32fdad4021a24cebe9ca13b1cd5a976340f62977d73c13
+size 1465

checkpoint-200/special_tokens_map.json

@@ -0,0 +1,30 @@
+{
+  "additional_special_tokens": [
+    "▁<PRE>",
+    "▁<MID>",
+    "▁<SUF>",
+    "▁<EOT>"
+  ],
+  "bos_token": {
+    "content": "<s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "</s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": "</s>",
+  "unk_token": {
+    "content": "<unk>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

checkpoint-200/tokenizer_config.json

@@ -0,0 +1,84 @@
+{
+  "add_bos_token": true,
+  "add_eos_token": false,
+  "added_tokens_decoder": {
+    "0": {
+      "content": "<unk>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "1": {
+      "content": "<s>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "2": {
+      "content": "</s>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "32007": {
+      "content": "▁<PRE>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "32008": {
+      "content": "▁<SUF>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "32009": {
+      "content": "▁<MID>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "32010": {
+      "content": "▁<EOT>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "additional_special_tokens": [
+    "▁<PRE>",
+    "▁<MID>",
+    "▁<SUF>",
+    "▁<EOT>"
+  ],
+  "bos_token": "<s>",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "</s>",
+  "eot_token": "▁<EOT>",
+  "extra_special_tokens": {},
+  "fill_token": "<FILL_ME>",
+  "legacy": null,
+  "middle_token": "▁<MID>",
+  "model_max_length": 1000000000000000019884624838656,
+  "pad_token": "</s>",
+  "prefix_token": "▁<PRE>",
+  "sp_model_kwargs": {},
+  "suffix_token": "▁<SUF>",
+  "tokenizer_class": "CodeLlamaTokenizer",
+  "unk_token": "<unk>",
+  "use_default_system_prompt": false
+}

checkpoint-200/trainer_state.json

@@ -0,0 +1,256 @@
+{
+  "best_global_step": null,
+  "best_metric": null,
+  "best_model_checkpoint": null,
+  "epoch": 0.6796941376380629,
+  "eval_steps": 100,
+  "global_step": 200,
+  "is_hyper_param_search": false,
+  "is_local_process_zero": true,
+  "is_world_process_zero": true,
+  "log_history": [
+    {
+      "entropy": 1.1900819569826127,
+      "epoch": 0.033984706881903144,
+      "grad_norm": 0.107421875,
+      "learning_rate": 6.666666666666667e-05,
+      "loss": 1.2022,
+      "mean_token_accuracy": 0.7261606067419052,
+      "num_tokens": 32329.0,
+      "step": 10
+    },
+    {
+      "entropy": 1.1123934835195541,
+      "epoch": 0.06796941376380629,
+      "grad_norm": 0.138671875,
+      "learning_rate": 0.00014074074074074076,
+      "loss": 1.1277,
+      "mean_token_accuracy": 0.7318252056837082,
+      "num_tokens": 67111.0,
+      "step": 20
+    },
+    {
+      "entropy": 1.0879137217998505,
+      "epoch": 0.10195412064570943,
+      "grad_norm": 0.3203125,
+      "learning_rate": 0.00019999731865174213,
+      "loss": 1.0625,
+      "mean_token_accuracy": 0.7478782564401627,
+      "num_tokens": 101352.0,
+      "step": 30
+    },
+    {
+      "entropy": 0.9986169628798962,
+      "epoch": 0.13593882752761258,
+      "grad_norm": 0.2890625,
+      "learning_rate": 0.00019990348656007261,
+      "loss": 1.0079,
+      "mean_token_accuracy": 0.7581020414829254,
+      "num_tokens": 133883.0,
+      "step": 40
+    },
+    {
+      "entropy": 1.05185257345438,
+      "epoch": 0.16992353440951571,
+      "grad_norm": 0.294921875,
+      "learning_rate": 0.00019967573081342103,
+      "loss": 1.0335,
+      "mean_token_accuracy": 0.7543211042881012,
+      "num_tokens": 168562.0,
+      "step": 50
+    },
+    {
+      "entropy": 0.9918051429092885,
+      "epoch": 0.20390824129141885,
+      "grad_norm": 0.263671875,
+      "learning_rate": 0.00019931435672527624,
+      "loss": 0.9977,
+      "mean_token_accuracy": 0.755571472644806,
+      "num_tokens": 202299.0,
+      "step": 60
+    },
+    {
+      "entropy": 0.8909615643322468,
+      "epoch": 0.23789294817332202,
+      "grad_norm": 0.298828125,
+      "learning_rate": 0.00019881984872856817,
+      "loss": 0.9168,
+      "mean_token_accuracy": 0.7724857151508331,
+      "num_tokens": 237132.0,
+      "step": 70
+    },
+    {
+      "entropy": 0.9534411959350109,
+      "epoch": 0.27187765505522515,
+      "grad_norm": 0.3203125,
+      "learning_rate": 0.00019819286972627066,
+      "loss": 0.9673,
+      "mean_token_accuracy": 0.7637621074914932,
+      "num_tokens": 271644.0,
+      "step": 80
+    },
+    {
+      "entropy": 0.9248364262282849,
+      "epoch": 0.3058623619371283,
+      "grad_norm": 0.26171875,
+      "learning_rate": 0.00019743426020275994,
+      "loss": 0.9077,
+      "mean_token_accuracy": 0.7676862999796867,
+      "num_tokens": 306939.0,
+      "step": 90
+    },
+    {
+      "entropy": 1.0025891564786433,
+      "epoch": 0.33984706881903143,
+      "grad_norm": 0.28125,
+      "learning_rate": 0.00019654503709711982,
+      "loss": 1.0295,
+      "mean_token_accuracy": 0.7536625176668167,
+      "num_tokens": 341079.0,
+      "step": 100
+    },
+    {
+      "epoch": 0.33984706881903143,
+      "eval_entropy": 0.9100993389175052,
+      "eval_loss": 0.9260265231132507,
+      "eval_mean_token_accuracy": 0.7760482584538103,
+      "eval_num_tokens": 341079.0,
+      "eval_runtime": 140.9825,
+      "eval_samples_per_second": 1.043,
+      "eval_steps_per_second": 1.043,
+      "step": 100
+    },
+    {
+      "entropy": 0.9492803812026978,
+      "epoch": 0.37383177570093457,
+      "grad_norm": 0.2578125,
+      "learning_rate": 0.00019552639243990402,
+      "loss": 0.9391,
+      "mean_token_accuracy": 0.7675268396735191,
+      "num_tokens": 374431.0,
+      "step": 110
+    },
+    {
+      "entropy": 0.86210857629776,
+      "epoch": 0.4078164825828377,
+      "grad_norm": 0.30078125,
+      "learning_rate": 0.00019437969175518295,
+      "loss": 0.8868,
+      "mean_token_accuracy": 0.7805617764592171,
+      "num_tokens": 410401.0,
+      "step": 120
+    },
+    {
+      "entropy": 0.9226490631699562,
+      "epoch": 0.44180118946474084,
+      "grad_norm": 0.3046875,
+      "learning_rate": 0.0001931064722300175,
+      "loss": 0.9118,
+      "mean_token_accuracy": 0.7770121544599533,
+      "num_tokens": 443280.0,
+      "step": 130
+    },
+    {
+      "entropy": 0.8999310165643692,
+      "epoch": 0.47578589634664403,
+      "grad_norm": 0.2734375,
+      "learning_rate": 0.00019170844065381285,
+      "loss": 0.9117,
+      "mean_token_accuracy": 0.7742907002568244,
+      "num_tokens": 478900.0,
+      "step": 140
+    },
+    {
+      "entropy": 0.8831319987773896,
+      "epoch": 0.5097706032285472,
+      "grad_norm": 0.255859375,
+      "learning_rate": 0.00019018747113031564,
+      "loss": 0.8893,
+      "mean_token_accuracy": 0.7864658862352372,
+      "num_tokens": 513687.0,
+      "step": 150
+    },
+    {
+      "entropy": 0.8636823572218418,
+      "epoch": 0.5437553101104503,
+      "grad_norm": 0.490234375,
+      "learning_rate": 0.000188545602565321,
+      "loss": 0.8824,
+      "mean_token_accuracy": 0.7811477512121201,
+      "num_tokens": 549223.0,
+      "step": 160
+    },
+    {
+      "entropy": 0.9621140897274018,
+      "epoch": 0.5777400169923534,
+      "grad_norm": 0.30859375,
+      "learning_rate": 0.00018678503593345754,
+      "loss": 0.959,
+      "mean_token_accuracy": 0.765991534292698,
+      "num_tokens": 582348.0,
+      "step": 170
+    },
+    {
+      "entropy": 0.9770274326205254,
+      "epoch": 0.6117247238742566,
+      "grad_norm": 0.345703125,
+      "learning_rate": 0.00018490813132771393,
+      "loss": 0.9698,
+      "mean_token_accuracy": 0.767307311296463,
+      "num_tokens": 614105.0,
+      "step": 180
+    },
+    {
+      "entropy": 0.8605956405401229,
+      "epoch": 0.6457094307561597,
+      "grad_norm": 0.408203125,
+      "learning_rate": 0.00018291740479566283,
+      "loss": 0.8559,
+      "mean_token_accuracy": 0.7881909653544426,
+      "num_tokens": 649045.0,
+      "step": 190
+    },
+    {
+      "entropy": 0.8583430543541908,
+      "epoch": 0.6796941376380629,
+      "grad_norm": 0.26953125,
+      "learning_rate": 0.00018081552496662258,
+      "loss": 0.8807,
+      "mean_token_accuracy": 0.7877280384302139,
+      "num_tokens": 681327.0,
+      "step": 200
+    },
+    {
+      "epoch": 0.6796941376380629,
+      "eval_entropy": 0.896664229785504,
+      "eval_loss": 0.89118492603302,
+      "eval_mean_token_accuracy": 0.7833221583139329,
+      "eval_num_tokens": 681327.0,
+      "eval_runtime": 140.9445,
+      "eval_samples_per_second": 1.043,
+      "eval_steps_per_second": 1.043,
+      "step": 200
+    }
+  ],
+  "logging_steps": 10,
+  "max_steps": 885,
+  "num_input_tokens_seen": 0,
+  "num_train_epochs": 3,
+  "save_steps": 100,
+  "stateful_callbacks": {
+    "TrainerControl": {
+      "args": {
+        "should_epoch_stop": false,
+        "should_evaluate": false,
+        "should_log": false,
+        "should_save": true,
+        "should_training_stop": false
+      },
+      "attributes": {}
+    }
+  },
+  "total_flos": 2.704512984256512e+16,
+  "train_batch_size": 1,
+  "trial_name": null,
+  "trial_params": null
+}

checkpoint-200/training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:be523aaac59d69ae5ec9c8576a8ed2c5b5e19240aee6056559023b48d8fa85f8
+size 6161

checkpoint-300/README.md

@@ -0,0 +1,209 @@
+---
+base_model: codellama/CodeLlama-7b-hf
+library_name: peft
+pipeline_tag: text-generation
+tags:
+- base_model:adapter:codellama/CodeLlama-7b-hf
+- lora
+- sft
+- transformers
+- trl
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.18.0

checkpoint-300/adapter_config.json

@@ -0,0 +1,41 @@
+{
+  "alora_invocation_tokens": null,
+  "alpha_pattern": {},
+  "arrow_config": null,
+  "auto_mapping": null,
+  "base_model_name_or_path": "codellama/CodeLlama-7b-hf",
+  "bias": "none",
+  "corda_config": null,
+  "ensure_weight_tying": false,
+  "eva_config": null,
+  "exclude_modules": null,
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 32,
+  "lora_bias": false,
+  "lora_dropout": 0.05,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "peft_version": "0.18.0",
+  "qalora_group_size": 16,
+  "r": 16,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "v_proj",
+    "q_proj"
+  ],
+  "target_parameters": null,
+  "task_type": "CAUSAL_LM",
+  "trainable_token_indices": null,
+  "use_dora": false,
+  "use_qalora": false,
+  "use_rslora": false
+}

checkpoint-300/adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:70c582e19a4946ff2f0ee7e0d20f3fffb8fb9ad63237e0f4d8706bda0a3caaab
+size 16794456

checkpoint-300/optimizer.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4dabad76c635404d0e438222a2100429fc25f5605ca8490915ccf53fe219aaef
+size 17192139

checkpoint-300/rng_state.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:36b11c1e37bc8bd075d0af84cdec85648ec8d5c7cbb3d72e1ff14568e6c860f0
+size 14645

checkpoint-300/scheduler.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5ef20fe686595564c1432a7f12c7bfd8a493080340a1bc4720458d76f2a95359
+size 1465

checkpoint-300/special_tokens_map.json

@@ -0,0 +1,30 @@
+{
+  "additional_special_tokens": [
+    "▁<PRE>",
+    "▁<MID>",
+    "▁<SUF>",
+    "▁<EOT>"
+  ],
+  "bos_token": {
+    "content": "<s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "</s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": "</s>",
+  "unk_token": {
+    "content": "<unk>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

checkpoint-300/tokenizer_config.json

@@ -0,0 +1,84 @@
+{
+  "add_bos_token": true,
+  "add_eos_token": false,
+  "added_tokens_decoder": {
+    "0": {
+      "content": "<unk>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "1": {
+      "content": "<s>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "2": {
+      "content": "</s>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "32007": {
+      "content": "▁<PRE>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "32008": {
+      "content": "▁<SUF>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "32009": {
+      "content": "▁<MID>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "32010": {
+      "content": "▁<EOT>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "additional_special_tokens": [
+    "▁<PRE>",
+    "▁<MID>",
+    "▁<SUF>",
+    "▁<EOT>"
+  ],
+  "bos_token": "<s>",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "</s>",
+  "eot_token": "▁<EOT>",
+  "extra_special_tokens": {},
+  "fill_token": "<FILL_ME>",
+  "legacy": null,
+  "middle_token": "▁<MID>",
+  "model_max_length": 1000000000000000019884624838656,
+  "pad_token": "</s>",
+  "prefix_token": "▁<PRE>",
+  "sp_model_kwargs": {},
+  "suffix_token": "▁<SUF>",
+  "tokenizer_class": "CodeLlamaTokenizer",
+  "unk_token": "<unk>",
+  "use_default_system_prompt": false
+}

checkpoint-300/trainer_state.json

@@ -0,0 +1,367 @@
+{
+  "best_global_step": null,
+  "best_metric": null,
+  "best_model_checkpoint": null,
+  "epoch": 1.0169923534409515,
+  "eval_steps": 100,
+  "global_step": 300,
+  "is_hyper_param_search": false,
+  "is_local_process_zero": true,
+  "is_world_process_zero": true,
+  "log_history": [
+    {
+      "entropy": 1.1900819569826127,
+      "epoch": 0.033984706881903144,
+      "grad_norm": 0.107421875,
+      "learning_rate": 6.666666666666667e-05,
+      "loss": 1.2022,
+      "mean_token_accuracy": 0.7261606067419052,
+      "num_tokens": 32329.0,
+      "step": 10
+    },
+    {
+      "entropy": 1.1123934835195541,
+      "epoch": 0.06796941376380629,
+      "grad_norm": 0.138671875,
+      "learning_rate": 0.00014074074074074076,
+      "loss": 1.1277,
+      "mean_token_accuracy": 0.7318252056837082,
+      "num_tokens": 67111.0,
+      "step": 20
+    },
+    {
+      "entropy": 1.0879137217998505,
+      "epoch": 0.10195412064570943,
+      "grad_norm": 0.3203125,
+      "learning_rate": 0.00019999731865174213,
+      "loss": 1.0625,
+      "mean_token_accuracy": 0.7478782564401627,
+      "num_tokens": 101352.0,
+      "step": 30
+    },
+    {
+      "entropy": 0.9986169628798962,
+      "epoch": 0.13593882752761258,
+      "grad_norm": 0.2890625,
+      "learning_rate": 0.00019990348656007261,
+      "loss": 1.0079,
+      "mean_token_accuracy": 0.7581020414829254,
+      "num_tokens": 133883.0,
+      "step": 40
+    },
+    {
+      "entropy": 1.05185257345438,
+      "epoch": 0.16992353440951571,
+      "grad_norm": 0.294921875,
+      "learning_rate": 0.00019967573081342103,
+      "loss": 1.0335,
+      "mean_token_accuracy": 0.7543211042881012,
+      "num_tokens": 168562.0,
+      "step": 50
+    },
+    {
+      "entropy": 0.9918051429092885,
+      "epoch": 0.20390824129141885,
+      "grad_norm": 0.263671875,
+      "learning_rate": 0.00019931435672527624,
+      "loss": 0.9977,
+      "mean_token_accuracy": 0.755571472644806,
+      "num_tokens": 202299.0,
+      "step": 60
+    },
+    {
+      "entropy": 0.8909615643322468,
+      "epoch": 0.23789294817332202,
+      "grad_norm": 0.298828125,
+      "learning_rate": 0.00019881984872856817,
+      "loss": 0.9168,
+      "mean_token_accuracy": 0.7724857151508331,
+      "num_tokens": 237132.0,
+      "step": 70
+    },
+    {
+      "entropy": 0.9534411959350109,
+      "epoch": 0.27187765505522515,
+      "grad_norm": 0.3203125,
+      "learning_rate": 0.00019819286972627066,
+      "loss": 0.9673,
+      "mean_token_accuracy": 0.7637621074914932,
+      "num_tokens": 271644.0,
+      "step": 80
+    },
+    {
+      "entropy": 0.9248364262282849,
+      "epoch": 0.3058623619371283,
+      "grad_norm": 0.26171875,
+      "learning_rate": 0.00019743426020275994,
+      "loss": 0.9077,
+      "mean_token_accuracy": 0.7676862999796867,
+      "num_tokens": 306939.0,
+      "step": 90
+    },
+    {
+      "entropy": 1.0025891564786433,
+      "epoch": 0.33984706881903143,
+      "grad_norm": 0.28125,
+      "learning_rate": 0.00019654503709711982,
+      "loss": 1.0295,
+      "mean_token_accuracy": 0.7536625176668167,
+      "num_tokens": 341079.0,
+      "step": 100
+    },
+    {
+      "epoch": 0.33984706881903143,
+      "eval_entropy": 0.9100993389175052,
+      "eval_loss": 0.9260265231132507,
+      "eval_mean_token_accuracy": 0.7760482584538103,
+      "eval_num_tokens": 341079.0,
+      "eval_runtime": 140.9825,
+      "eval_samples_per_second": 1.043,
+      "eval_steps_per_second": 1.043,
+      "step": 100
+    },
+    {
+      "entropy": 0.9492803812026978,
+      "epoch": 0.37383177570093457,
+      "grad_norm": 0.2578125,
+      "learning_rate": 0.00019552639243990402,
+      "loss": 0.9391,
+      "mean_token_accuracy": 0.7675268396735191,
+      "num_tokens": 374431.0,
+      "step": 110
+    },
+    {
+      "entropy": 0.86210857629776,
+      "epoch": 0.4078164825828377,
+      "grad_norm": 0.30078125,
+      "learning_rate": 0.00019437969175518295,
+      "loss": 0.8868,
+      "mean_token_accuracy": 0.7805617764592171,
+      "num_tokens": 410401.0,
+      "step": 120
+    },
+    {
+      "entropy": 0.9226490631699562,
+      "epoch": 0.44180118946474084,
+      "grad_norm": 0.3046875,
+      "learning_rate": 0.0001931064722300175,
+      "loss": 0.9118,
+      "mean_token_accuracy": 0.7770121544599533,
+      "num_tokens": 443280.0,
+      "step": 130
+    },
+    {
+      "entropy": 0.8999310165643692,
+      "epoch": 0.47578589634664403,
+      "grad_norm": 0.2734375,
+      "learning_rate": 0.00019170844065381285,
+      "loss": 0.9117,
+      "mean_token_accuracy": 0.7742907002568244,
+      "num_tokens": 478900.0,
+      "step": 140
+    },
+    {
+      "entropy": 0.8831319987773896,
+      "epoch": 0.5097706032285472,
+      "grad_norm": 0.255859375,
+      "learning_rate": 0.00019018747113031564,
+      "loss": 0.8893,
+      "mean_token_accuracy": 0.7864658862352372,
+      "num_tokens": 513687.0,
+      "step": 150
+    },
+    {
+      "entropy": 0.8636823572218418,
+      "epoch": 0.5437553101104503,
+      "grad_norm": 0.490234375,
+      "learning_rate": 0.000188545602565321,
+      "loss": 0.8824,
+      "mean_token_accuracy": 0.7811477512121201,
+      "num_tokens": 549223.0,
+      "step": 160
+    },
+    {
+      "entropy": 0.9621140897274018,
+      "epoch": 0.5777400169923534,
+      "grad_norm": 0.30859375,
+      "learning_rate": 0.00018678503593345754,
+      "loss": 0.959,
+      "mean_token_accuracy": 0.765991534292698,
+      "num_tokens": 582348.0,
+      "step": 170
+    },
+    {
+      "entropy": 0.9770274326205254,
+      "epoch": 0.6117247238742566,
+      "grad_norm": 0.345703125,
+      "learning_rate": 0.00018490813132771393,
+      "loss": 0.9698,
+      "mean_token_accuracy": 0.767307311296463,
+      "num_tokens": 614105.0,
+      "step": 180
+    },
+    {
+      "entropy": 0.8605956405401229,
+      "epoch": 0.6457094307561597,
+      "grad_norm": 0.408203125,
+      "learning_rate": 0.00018291740479566283,
+      "loss": 0.8559,
+      "mean_token_accuracy": 0.7881909653544426,
+      "num_tokens": 649045.0,
+      "step": 190
+    },
+    {
+      "entropy": 0.8583430543541908,
+      "epoch": 0.6796941376380629,
+      "grad_norm": 0.26953125,
+      "learning_rate": 0.00018081552496662258,
+      "loss": 0.8807,
+      "mean_token_accuracy": 0.7877280384302139,
+      "num_tokens": 681327.0,
+      "step": 200
+    },
+    {
+      "epoch": 0.6796941376380629,
+      "eval_entropy": 0.896664229785504,
+      "eval_loss": 0.89118492603302,
+      "eval_mean_token_accuracy": 0.7833221583139329,
+      "eval_num_tokens": 681327.0,
+      "eval_runtime": 140.9445,
+      "eval_samples_per_second": 1.043,
+      "eval_steps_per_second": 1.043,
+      "step": 200
+    },
+    {
+      "entropy": 0.9148187682032585,
+      "epoch": 0.713678844519966,
+      "grad_norm": 0.287109375,
+      "learning_rate": 0.00017860530947427875,
+      "loss": 0.9274,
+      "mean_token_accuracy": 0.7749700739979744,
+      "num_tokens": 714260.0,
+      "step": 210
+    },
+    {
+      "entropy": 0.923950819671154,
+      "epoch": 0.7476635514018691,
+      "grad_norm": 0.32421875,
+      "learning_rate": 0.0001762897211795607,
+      "loss": 0.8731,
+      "mean_token_accuracy": 0.7750329807400703,
+      "num_tokens": 746221.0,
+      "step": 220
+    },
+    {
+      "entropy": 1.0102886088192462,
+      "epoch": 0.7816482582837723,
+      "grad_norm": 0.330078125,
+      "learning_rate": 0.0001738718641988365,
+      "loss": 1.0153,
+      "mean_token_accuracy": 0.7520556971430779,
+      "num_tokens": 781925.0,
+      "step": 230
+    },
+    {
+      "entropy": 0.8034560449421406,
+      "epoch": 0.8156329651656754,
+      "grad_norm": 0.32421875,
+      "learning_rate": 0.00017135497974275088,
+      "loss": 0.8024,
+      "mean_token_accuracy": 0.8065776988863945,
+      "num_tokens": 819257.0,
+      "step": 240
+    },
+    {
+      "entropy": 0.8587650842964649,
+      "epoch": 0.8496176720475785,
+      "grad_norm": 0.244140625,
+      "learning_rate": 0.00016874244177128396,
+      "loss": 0.8729,
+      "mean_token_accuracy": 0.7839998126029968,
+      "num_tokens": 854177.0,
+      "step": 250
+    },
+    {
+      "entropy": 0.8471586465835571,
+      "epoch": 0.8836023789294817,
+      "grad_norm": 0.333984375,
+      "learning_rate": 0.00016603775247085546,
+      "loss": 0.8693,
+      "mean_token_accuracy": 0.7884811326861382,
+      "num_tokens": 887426.0,
+      "step": 260
+    },
+    {
+      "entropy": 0.8740401305258274,
+      "epoch": 0.9175870858113849,
+      "grad_norm": 0.3125,
+      "learning_rate": 0.00016324453755953773,
+      "loss": 0.881,
+      "mean_token_accuracy": 0.787057913839817,
+      "num_tokens": 918169.0,
+      "step": 270
+    },
+    {
+      "entropy": 0.8283644251525402,
+      "epoch": 0.9515717926932881,
+      "grad_norm": 0.32421875,
+      "learning_rate": 0.00016036654142667043,
+      "loss": 0.8293,
+      "mean_token_accuracy": 0.7984860435128212,
+      "num_tokens": 953966.0,
+      "step": 280
+    },
+    {
+      "entropy": 0.8554800219833851,
+      "epoch": 0.9855564995751912,
+      "grad_norm": 0.40234375,
+      "learning_rate": 0.00015740762211339314,
+      "loss": 0.8646,
+      "mean_token_accuracy": 0.7841584324836731,
+      "num_tokens": 986605.0,
+      "step": 290
+    },
+    {
+      "entropy": 0.8761118473233404,
+      "epoch": 1.0169923534409515,
+      "grad_norm": 0.29296875,
+      "learning_rate": 0.00015437174614082416,
+      "loss": 0.9419,
+      "mean_token_accuracy": 0.7891149762514476,
+      "num_tokens": 1018372.0,
+      "step": 300
+    },
+    {
+      "epoch": 1.0169923534409515,
+      "eval_entropy": 0.8560160027474774,
+      "eval_loss": 0.8672717809677124,
+      "eval_mean_token_accuracy": 0.7890248249988167,
+      "eval_num_tokens": 1018372.0,
+      "eval_runtime": 140.8836,
+      "eval_samples_per_second": 1.043,
+      "eval_steps_per_second": 1.043,
+      "step": 300
+    }
+  ],
+  "logging_steps": 10,
+  "max_steps": 885,
+  "num_input_tokens_seen": 0,
+  "num_train_epochs": 3,
+  "save_steps": 100,
+  "stateful_callbacks": {
+    "TrainerControl": {
+      "args": {
+        "should_epoch_stop": false,
+        "should_evaluate": false,
+        "should_log": false,
+        "should_save": true,
+        "should_training_stop": false
+      },
+      "attributes": {}
+    }
+  },
+  "total_flos": 4.042405917868032e+16,
+  "train_batch_size": 1,
+  "trial_name": null,
+  "trial_params": null
+}

checkpoint-300/training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:be523aaac59d69ae5ec9c8576a8ed2c5b5e19240aee6056559023b48d8fa85f8
+size 6161

checkpoint-400/README.md

@@ -0,0 +1,209 @@
+---
+base_model: codellama/CodeLlama-7b-hf
+library_name: peft
+pipeline_tag: text-generation
+tags:
+- base_model:adapter:codellama/CodeLlama-7b-hf
+- lora
+- sft
+- transformers
+- trl
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.18.0

checkpoint-400/adapter_config.json

@@ -0,0 +1,41 @@
+{
+  "alora_invocation_tokens": null,
+  "alpha_pattern": {},
+  "arrow_config": null,
+  "auto_mapping": null,
+  "base_model_name_or_path": "codellama/CodeLlama-7b-hf",
+  "bias": "none",
+  "corda_config": null,
+  "ensure_weight_tying": false,
+  "eva_config": null,
+  "exclude_modules": null,
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 32,
+  "lora_bias": false,
+  "lora_dropout": 0.05,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "peft_version": "0.18.0",
+  "qalora_group_size": 16,
+  "r": 16,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "v_proj",
+    "q_proj"
+  ],
+  "target_parameters": null,
+  "task_type": "CAUSAL_LM",
+  "trainable_token_indices": null,
+  "use_dora": false,
+  "use_qalora": false,
+  "use_rslora": false
+}

checkpoint-400/adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d966cbb5adfc3a289dc38d4916aa4ce7d02cf9b787e35f42c2947e494ff77ca0
+size 16794456

checkpoint-400/optimizer.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0b4acfa30b0b6880b3cd15d1bf6e557d5ac23145b7e803be18c9fcdcbe7ebe0e
+size 17192139

checkpoint-400/rng_state.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:86a634bdf900caeadc121395ee5a65839dcdb8774fbebe092b7068f4e3be6b9e
+size 14645

checkpoint-400/scheduler.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:36c5a88c915ba7053aed812fb6f13ed9035d0b1069320b12dd5e0e7ad07073e4
+size 1465

checkpoint-400/special_tokens_map.json

@@ -0,0 +1,30 @@
+{
+  "additional_special_tokens": [
+    "▁<PRE>",
+    "▁<MID>",
+    "▁<SUF>",
+    "▁<EOT>"
+  ],
+  "bos_token": {
+    "content": "<s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "</s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": "</s>",
+  "unk_token": {
+    "content": "<unk>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

checkpoint-400/tokenizer_config.json

@@ -0,0 +1,84 @@
+{
+  "add_bos_token": true,
+  "add_eos_token": false,
+  "added_tokens_decoder": {
+    "0": {
+      "content": "<unk>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "1": {
+      "content": "<s>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "2": {
+      "content": "</s>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "32007": {
+      "content": "▁<PRE>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "32008": {
+      "content": "▁<SUF>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "32009": {
+      "content": "▁<MID>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "32010": {
+      "content": "▁<EOT>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "additional_special_tokens": [
+    "▁<PRE>",
+    "▁<MID>",
+    "▁<SUF>",
+    "▁<EOT>"
+  ],
+  "bos_token": "<s>",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "</s>",
+  "eot_token": "▁<EOT>",
+  "extra_special_tokens": {},
+  "fill_token": "<FILL_ME>",
+  "legacy": null,
+  "middle_token": "▁<MID>",
+  "model_max_length": 1000000000000000019884624838656,
+  "pad_token": "</s>",
+  "prefix_token": "▁<PRE>",
+  "sp_model_kwargs": {},
+  "suffix_token": "▁<SUF>",
+  "tokenizer_class": "CodeLlamaTokenizer",
+  "unk_token": "<unk>",
+  "use_default_system_prompt": false
+}

checkpoint-400/trainer_state.json

@@ -0,0 +1,478 @@
+{
+  "best_global_step": null,
+  "best_metric": null,
+  "best_model_checkpoint": null,
+  "epoch": 1.3568394222599829,
+  "eval_steps": 100,
+  "global_step": 400,
+  "is_hyper_param_search": false,
+  "is_local_process_zero": true,
+  "is_world_process_zero": true,
+  "log_history": [
+    {
+      "entropy": 1.1900819569826127,
+      "epoch": 0.033984706881903144,
+      "grad_norm": 0.107421875,
+      "learning_rate": 6.666666666666667e-05,
+      "loss": 1.2022,
+      "mean_token_accuracy": 0.7261606067419052,
+      "num_tokens": 32329.0,
+      "step": 10
+    },
+    {
+      "entropy": 1.1123934835195541,
+      "epoch": 0.06796941376380629,
+      "grad_norm": 0.138671875,
+      "learning_rate": 0.00014074074074074076,
+      "loss": 1.1277,
+      "mean_token_accuracy": 0.7318252056837082,
+      "num_tokens": 67111.0,
+      "step": 20
+    },
+    {
+      "entropy": 1.0879137217998505,
+      "epoch": 0.10195412064570943,
+      "grad_norm": 0.3203125,
+      "learning_rate": 0.00019999731865174213,
+      "loss": 1.0625,
+      "mean_token_accuracy": 0.7478782564401627,
+      "num_tokens": 101352.0,
+      "step": 30
+    },
+    {
+      "entropy": 0.9986169628798962,
+      "epoch": 0.13593882752761258,
+      "grad_norm": 0.2890625,
+      "learning_rate": 0.00019990348656007261,
+      "loss": 1.0079,
+      "mean_token_accuracy": 0.7581020414829254,
+      "num_tokens": 133883.0,
+      "step": 40
+    },
+    {
+      "entropy": 1.05185257345438,
+      "epoch": 0.16992353440951571,
+      "grad_norm": 0.294921875,
+      "learning_rate": 0.00019967573081342103,
+      "loss": 1.0335,
+      "mean_token_accuracy": 0.7543211042881012,
+      "num_tokens": 168562.0,
+      "step": 50
+    },
+    {
+      "entropy": 0.9918051429092885,
+      "epoch": 0.20390824129141885,
+      "grad_norm": 0.263671875,
+      "learning_rate": 0.00019931435672527624,
+      "loss": 0.9977,
+      "mean_token_accuracy": 0.755571472644806,
+      "num_tokens": 202299.0,
+      "step": 60
+    },
+    {
+      "entropy": 0.8909615643322468,
+      "epoch": 0.23789294817332202,
+      "grad_norm": 0.298828125,
+      "learning_rate": 0.00019881984872856817,
+      "loss": 0.9168,
+      "mean_token_accuracy": 0.7724857151508331,
+      "num_tokens": 237132.0,
+      "step": 70
+    },
+    {
+      "entropy": 0.9534411959350109,
+      "epoch": 0.27187765505522515,
+      "grad_norm": 0.3203125,
+      "learning_rate": 0.00019819286972627066,
+      "loss": 0.9673,
+      "mean_token_accuracy": 0.7637621074914932,
+      "num_tokens": 271644.0,
+      "step": 80
+    },
+    {
+      "entropy": 0.9248364262282849,
+      "epoch": 0.3058623619371283,
+      "grad_norm": 0.26171875,
+      "learning_rate": 0.00019743426020275994,
+      "loss": 0.9077,
+      "mean_token_accuracy": 0.7676862999796867,
+      "num_tokens": 306939.0,
+      "step": 90
+    },
+    {
+      "entropy": 1.0025891564786433,
+      "epoch": 0.33984706881903143,
+      "grad_norm": 0.28125,
+      "learning_rate": 0.00019654503709711982,
+      "loss": 1.0295,
+      "mean_token_accuracy": 0.7536625176668167,
+      "num_tokens": 341079.0,
+      "step": 100
+    },
+    {
+      "epoch": 0.33984706881903143,
+      "eval_entropy": 0.9100993389175052,
+      "eval_loss": 0.9260265231132507,
+      "eval_mean_token_accuracy": 0.7760482584538103,
+      "eval_num_tokens": 341079.0,
+      "eval_runtime": 140.9825,
+      "eval_samples_per_second": 1.043,
+      "eval_steps_per_second": 1.043,
+      "step": 100
+    },
+    {
+      "entropy": 0.9492803812026978,
+      "epoch": 0.37383177570093457,
+      "grad_norm": 0.2578125,
+      "learning_rate": 0.00019552639243990402,
+      "loss": 0.9391,
+      "mean_token_accuracy": 0.7675268396735191,
+      "num_tokens": 374431.0,
+      "step": 110
+    },
+    {
+      "entropy": 0.86210857629776,
+      "epoch": 0.4078164825828377,
+      "grad_norm": 0.30078125,
+      "learning_rate": 0.00019437969175518295,
+      "loss": 0.8868,
+      "mean_token_accuracy": 0.7805617764592171,
+      "num_tokens": 410401.0,
+      "step": 120
+    },
+    {
+      "entropy": 0.9226490631699562,
+      "epoch": 0.44180118946474084,
+      "grad_norm": 0.3046875,
+      "learning_rate": 0.0001931064722300175,
+      "loss": 0.9118,
+      "mean_token_accuracy": 0.7770121544599533,
+      "num_tokens": 443280.0,
+      "step": 130
+    },
+    {
+      "entropy": 0.8999310165643692,
+      "epoch": 0.47578589634664403,
+      "grad_norm": 0.2734375,
+      "learning_rate": 0.00019170844065381285,
+      "loss": 0.9117,
+      "mean_token_accuracy": 0.7742907002568244,
+      "num_tokens": 478900.0,
+      "step": 140
+    },
+    {
+      "entropy": 0.8831319987773896,
+      "epoch": 0.5097706032285472,
+      "grad_norm": 0.255859375,
+      "learning_rate": 0.00019018747113031564,
+      "loss": 0.8893,
+      "mean_token_accuracy": 0.7864658862352372,
+      "num_tokens": 513687.0,
+      "step": 150
+    },
+    {
+      "entropy": 0.8636823572218418,
+      "epoch": 0.5437553101104503,
+      "grad_norm": 0.490234375,
+      "learning_rate": 0.000188545602565321,
+      "loss": 0.8824,
+      "mean_token_accuracy": 0.7811477512121201,
+      "num_tokens": 549223.0,
+      "step": 160
+    },
+    {
+      "entropy": 0.9621140897274018,
+      "epoch": 0.5777400169923534,
+      "grad_norm": 0.30859375,
+      "learning_rate": 0.00018678503593345754,
+      "loss": 0.959,
+      "mean_token_accuracy": 0.765991534292698,
+      "num_tokens": 582348.0,
+      "step": 170
+    },
+    {
+      "entropy": 0.9770274326205254,
+      "epoch": 0.6117247238742566,
+      "grad_norm": 0.345703125,
+      "learning_rate": 0.00018490813132771393,
+      "loss": 0.9698,
+      "mean_token_accuracy": 0.767307311296463,
+      "num_tokens": 614105.0,
+      "step": 180
+    },
+    {
+      "entropy": 0.8605956405401229,
+      "epoch": 0.6457094307561597,
+      "grad_norm": 0.408203125,
+      "learning_rate": 0.00018291740479566283,
+      "loss": 0.8559,
+      "mean_token_accuracy": 0.7881909653544426,
+      "num_tokens": 649045.0,
+      "step": 190
+    },
+    {
+      "entropy": 0.8583430543541908,
+      "epoch": 0.6796941376380629,
+      "grad_norm": 0.26953125,
+      "learning_rate": 0.00018081552496662258,
+      "loss": 0.8807,
+      "mean_token_accuracy": 0.7877280384302139,
+      "num_tokens": 681327.0,
+      "step": 200
+    },
+    {
+      "epoch": 0.6796941376380629,
+      "eval_entropy": 0.896664229785504,
+      "eval_loss": 0.89118492603302,
+      "eval_mean_token_accuracy": 0.7833221583139329,
+      "eval_num_tokens": 681327.0,
+      "eval_runtime": 140.9445,
+      "eval_samples_per_second": 1.043,
+      "eval_steps_per_second": 1.043,
+      "step": 200
+    },
+    {
+      "entropy": 0.9148187682032585,
+      "epoch": 0.713678844519966,
+      "grad_norm": 0.287109375,
+      "learning_rate": 0.00017860530947427875,
+      "loss": 0.9274,
+      "mean_token_accuracy": 0.7749700739979744,
+      "num_tokens": 714260.0,
+      "step": 210
+    },
+    {
+      "entropy": 0.923950819671154,
+      "epoch": 0.7476635514018691,
+      "grad_norm": 0.32421875,
+      "learning_rate": 0.0001762897211795607,
+      "loss": 0.8731,
+      "mean_token_accuracy": 0.7750329807400703,
+      "num_tokens": 746221.0,
+      "step": 220
+    },
+    {
+      "entropy": 1.0102886088192462,
+      "epoch": 0.7816482582837723,
+      "grad_norm": 0.330078125,
+      "learning_rate": 0.0001738718641988365,
+      "loss": 1.0153,
+      "mean_token_accuracy": 0.7520556971430779,
+      "num_tokens": 781925.0,
+      "step": 230
+    },
+    {
+      "entropy": 0.8034560449421406,
+      "epoch": 0.8156329651656754,
+      "grad_norm": 0.32421875,
+      "learning_rate": 0.00017135497974275088,
+      "loss": 0.8024,
+      "mean_token_accuracy": 0.8065776988863945,
+      "num_tokens": 819257.0,
+      "step": 240
+    },
+    {
+      "entropy": 0.8587650842964649,
+      "epoch": 0.8496176720475785,
+      "grad_norm": 0.244140625,
+      "learning_rate": 0.00016874244177128396,
+      "loss": 0.8729,
+      "mean_token_accuracy": 0.7839998126029968,
+      "num_tokens": 854177.0,
+      "step": 250
+    },
+    {
+      "entropy": 0.8471586465835571,
+      "epoch": 0.8836023789294817,
+      "grad_norm": 0.333984375,
+      "learning_rate": 0.00016603775247085546,
+      "loss": 0.8693,
+      "mean_token_accuracy": 0.7884811326861382,
+      "num_tokens": 887426.0,
+      "step": 260
+    },
+    {
+      "entropy": 0.8740401305258274,
+      "epoch": 0.9175870858113849,
+      "grad_norm": 0.3125,
+      "learning_rate": 0.00016324453755953773,
+      "loss": 0.881,
+      "mean_token_accuracy": 0.787057913839817,
+      "num_tokens": 918169.0,
+      "step": 270
+    },
+    {
+      "entropy": 0.8283644251525402,
+      "epoch": 0.9515717926932881,
+      "grad_norm": 0.32421875,
+      "learning_rate": 0.00016036654142667043,
+      "loss": 0.8293,
+      "mean_token_accuracy": 0.7984860435128212,
+      "num_tokens": 953966.0,
+      "step": 280
+    },
+    {
+      "entropy": 0.8554800219833851,
+      "epoch": 0.9855564995751912,
+      "grad_norm": 0.40234375,
+      "learning_rate": 0.00015740762211339314,
+      "loss": 0.8646,
+      "mean_token_accuracy": 0.7841584324836731,
+      "num_tokens": 986605.0,
+      "step": 290
+    },
+    {
+      "entropy": 0.8761118473233404,
+      "epoch": 1.0169923534409515,
+      "grad_norm": 0.29296875,
+      "learning_rate": 0.00015437174614082416,
+      "loss": 0.9419,
+      "mean_token_accuracy": 0.7891149762514476,
+      "num_tokens": 1018372.0,
+      "step": 300
+    },
+    {
+      "epoch": 1.0169923534409515,
+      "eval_entropy": 0.8560160027474774,
+      "eval_loss": 0.8672717809677124,
+      "eval_mean_token_accuracy": 0.7890248249988167,
+      "eval_num_tokens": 1018372.0,
+      "eval_runtime": 140.8836,
+      "eval_samples_per_second": 1.043,
+      "eval_steps_per_second": 1.043,
+      "step": 300
+    },
+    {
+      "entropy": 0.835958081483841,
+      "epoch": 1.0509770603228548,
+      "grad_norm": 0.3515625,
+      "learning_rate": 0.00015126298319281857,
+      "loss": 0.8349,
+      "mean_token_accuracy": 0.7921741649508476,
+      "num_tokens": 1053253.0,
+      "step": 310
+    },
+    {
+      "entropy": 0.8159842237830162,
+      "epoch": 1.0849617672047578,
+      "grad_norm": 0.46875,
+      "learning_rate": 0.00014808550066043352,
+      "loss": 0.8036,
+      "mean_token_accuracy": 0.7972497373819352,
+      "num_tokens": 1087084.0,
+      "step": 320
+    },
+    {
+      "entropy": 0.8521868549287319,
+      "epoch": 1.118946474086661,
+      "grad_norm": 0.330078125,
+      "learning_rate": 0.00014484355805541413,
+      "loss": 0.9106,
+      "mean_token_accuracy": 0.7848088085651398,
+      "num_tokens": 1121833.0,
+      "step": 330
+    },
+    {
+      "entropy": 0.8146591357886791,
+      "epoch": 1.152931180968564,
+      "grad_norm": 0.412109375,
+      "learning_rate": 0.00014154150130018866,
+      "loss": 0.7855,
+      "mean_token_accuracy": 0.7983845800161362,
+      "num_tokens": 1155315.0,
+      "step": 340
+    },
+    {
+      "entropy": 0.8469755969941616,
+      "epoch": 1.1869158878504673,
+      "grad_norm": 0.455078125,
+      "learning_rate": 0.00013818375690202774,
+      "loss": 0.8677,
+      "mean_token_accuracy": 0.7855840787291527,
+      "num_tokens": 1188884.0,
+      "step": 350
+    },
+    {
+      "entropy": 0.8589501097798348,
+      "epoch": 1.2209005947323703,
+      "grad_norm": 0.38671875,
+      "learning_rate": 0.000134774826019177,
+      "loss": 0.8452,
+      "mean_token_accuracy": 0.786843791604042,
+      "num_tokens": 1223117.0,
+      "step": 360
+    },
+    {
+      "entropy": 0.8148165218532085,
+      "epoch": 1.2548853016142736,
+      "grad_norm": 0.365234375,
+      "learning_rate": 0.0001313192784269179,
+      "loss": 0.7977,
+      "mean_token_accuracy": 0.7989379152655601,
+      "num_tokens": 1257970.0,
+      "step": 370
+    },
+    {
+      "entropy": 0.8345869470387697,
+      "epoch": 1.2888700084961768,
+      "grad_norm": 0.37109375,
+      "learning_rate": 0.0001278217463916453,
+      "loss": 0.8308,
+      "mean_token_accuracy": 0.7944282591342926,
+      "num_tokens": 1294426.0,
+      "step": 380
+    },
+    {
+      "entropy": 0.8046329416334629,
+      "epoch": 1.3228547153780799,
+      "grad_norm": 0.49609375,
+      "learning_rate": 0.00012428691846117372,
+      "loss": 0.7869,
+      "mean_token_accuracy": 0.8005647033452987,
+      "num_tokens": 1327540.0,
+      "step": 390
+    },
+    {
+      "entropy": 0.8379782371222972,
+      "epoch": 1.3568394222599829,
+      "grad_norm": 0.4375,
+      "learning_rate": 0.00012071953317959692,
+      "loss": 0.8281,
+      "mean_token_accuracy": 0.7926659971475601,
+      "num_tokens": 1361518.0,
+      "step": 400
+    },
+    {
+      "epoch": 1.3568394222599829,
+      "eval_entropy": 0.8221898901016533,
+      "eval_loss": 0.8522771000862122,
+      "eval_mean_token_accuracy": 0.7922491832655303,
+      "eval_num_tokens": 1361518.0,
+      "eval_runtime": 140.9731,
+      "eval_samples_per_second": 1.043,
+      "eval_steps_per_second": 1.043,
+      "step": 400
+    }
+  ],
+  "logging_steps": 10,
+  "max_steps": 885,
+  "num_input_tokens_seen": 0,
+  "num_train_epochs": 3,
+  "save_steps": 100,
+  "stateful_callbacks": {
+    "TrainerControl": {
+      "args": {
+        "should_epoch_stop": false,
+        "should_evaluate": false,
+        "should_log": false,
+        "should_save": true,
+        "should_training_stop": false
+      },
+      "attributes": {}
+    }
+  },
+  "total_flos": 5.404516640759808e+16,
+  "train_batch_size": 1,
+  "trial_name": null,
+  "trial_params": null
+}

checkpoint-400/training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:be523aaac59d69ae5ec9c8576a8ed2c5b5e19240aee6056559023b48d8fa85f8
+size 6161

checkpoint-500/README.md

@@ -0,0 +1,209 @@
+---
+base_model: codellama/CodeLlama-7b-hf
+library_name: peft
+pipeline_tag: text-generation
+tags:
+- base_model:adapter:codellama/CodeLlama-7b-hf
+- lora
+- sft
+- transformers
+- trl
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.18.0

checkpoint-500/adapter_config.json

@@ -0,0 +1,41 @@
+{
+  "alora_invocation_tokens": null,
+  "alpha_pattern": {},
+  "arrow_config": null,
+  "auto_mapping": null,
+  "base_model_name_or_path": "codellama/CodeLlama-7b-hf",
+  "bias": "none",
+  "corda_config": null,
+  "ensure_weight_tying": false,
+  "eva_config": null,
+  "exclude_modules": null,
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 32,
+  "lora_bias": false,
+  "lora_dropout": 0.05,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "peft_version": "0.18.0",
+  "qalora_group_size": 16,
+  "r": 16,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "v_proj",
+    "q_proj"
+  ],
+  "target_parameters": null,
+  "task_type": "CAUSAL_LM",
+  "trainable_token_indices": null,
+  "use_dora": false,
+  "use_qalora": false,
+  "use_rslora": false
+}

checkpoint-500/adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:45bf22dc7e0afe79cfb94c2a15f2d57290f2123efc4f05037784ac056e65695a
+size 16794456