Upload folder using huggingface_hub

.gitattributes

@@ -33,3 +33,9 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+checkpoint-100/tokenizer.json filter=lfs diff=lfs merge=lfs -text
+checkpoint-200/tokenizer.json filter=lfs diff=lfs merge=lfs -text
+checkpoint-300/tokenizer.json filter=lfs diff=lfs merge=lfs -text
+checkpoint-400/tokenizer.json filter=lfs diff=lfs merge=lfs -text
+checkpoint-500/tokenizer.json filter=lfs diff=lfs merge=lfs -text
+tokenizer.json filter=lfs diff=lfs merge=lfs -text

README.md

@@ -1,3 +1,62 @@
----
-license: mit
----
+---
+base_model: unsloth/llama-3-8b-Instruct-bnb-4bit
+library_name: peft
+model_name: tutor_model_output
+tags:
+- base_model:adapter:unsloth/llama-3-8b-Instruct-bnb-4bit
+- lora
+- sft
+- transformers
+- trl
+licence: license
+pipeline_tag: text-generation
+---
+
+# Model Card for tutor_model_output
+
+This model is a fine-tuned version of [unsloth/llama-3-8b-Instruct-bnb-4bit](https://huggingface.co/unsloth/llama-3-8b-Instruct-bnb-4bit).
+It has been trained using [TRL](https://github.com/huggingface/trl).
+
+## Quick start
+
+```python
+from transformers import pipeline
+
+question = "If you had a time machine, but could only go to the past or the future once and never return, which would you choose and why?"
+generator = pipeline("text-generation", model="None", device="cuda")
+output = generator([{"role": "user", "content": question}], max_new_tokens=128, return_full_text=False)[0]
+print(output["generated_text"])
+```
+
+## Training procedure
+
+[<img src="https://raw.githubusercontent.com/wandb/assets/main/wandb-github-badge-28.svg" alt="Visualize in Weights & Biases" width="150" height="24"/>](https://wandb.ai/phgrouptechs-phgroup-technology-solutions-co-ltd/my-awesome-project/runs/7vqgjc2u) 
+
+
+
+This model was trained with SFT.
+
+### Framework versions
+
+- PEFT 0.18.1
+- TRL: 0.29.0
+- Transformers: 5.2.0
+- Pytorch: 2.8.0+cu128
+- Datasets: 4.6.0
+- Tokenizers: 0.22.2
+
+## Citations
+
+
+
+Cite TRL as:
+    
+```bibtex
+@software{vonwerra2020trl,
+  title   = {{TRL: Transformers Reinforcement Learning}},
+  author  = {von Werra, Leandro and Belkada, Younes and Tunstall, Lewis and Beeching, Edward and Thrush, Tristan and Lambert, Nathan and Huang, Shengyi and Rasul, Kashif and Gallouédec, Quentin},
+  license = {Apache-2.0},
+  url     = {https://github.com/huggingface/trl},
+  year    = {2020}
+}
+```

adapter_config.json

@@ -0,0 +1,46 @@
+{
+  "alora_invocation_tokens": null,
+  "alpha_pattern": {},
+  "arrow_config": null,
+  "auto_mapping": null,
+  "base_model_name_or_path": "unsloth/llama-3-8b-Instruct-bnb-4bit",
+  "bias": "none",
+  "corda_config": null,
+  "ensure_weight_tying": false,
+  "eva_config": null,
+  "exclude_modules": null,
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 32,
+  "lora_bias": false,
+  "lora_dropout": 0.05,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "peft_version": "0.18.1",
+  "qalora_group_size": 16,
+  "r": 16,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "down_proj",
+    "o_proj",
+    "k_proj",
+    "q_proj",
+    "v_proj",
+    "gate_proj",
+    "up_proj"
+  ],
+  "target_parameters": null,
+  "task_type": "CAUSAL_LM",
+  "trainable_token_indices": null,
+  "use_dora": false,
+  "use_qalora": false,
+  "use_rslora": false
+}

adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:27e2e812aa91b0af98fa9af3f5cbd95f3212af35d91ec3ab0e8d1cf1f47b5ba6
+size 83946192

chat_template.jinja

@@ -0,0 +1,5 @@
+{% set loop_messages = messages %}{% for message in loop_messages %}{% set content = '<|start_header_id|>' + message['role'] + '<|end_header_id|>
+
+'+ message['content'] | trim + '<|eot_id|>' %}{% if loop.index0 == 0 %}{% set content = bos_token + content %}{% endif %}{{ content }}{% endfor %}{% if add_generation_prompt %}{{ '<|start_header_id|>assistant<|end_header_id|>
+
+' }}{% endif %}

checkpoint-100/README.md

@@ -0,0 +1,209 @@
+---
+base_model: unsloth/llama-3-8b-Instruct-bnb-4bit
+library_name: peft
+pipeline_tag: text-generation
+tags:
+- base_model:adapter:unsloth/llama-3-8b-Instruct-bnb-4bit
+- lora
+- sft
+- transformers
+- trl
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.18.1

checkpoint-100/adapter_config.json

@@ -0,0 +1,46 @@
+{
+  "alora_invocation_tokens": null,
+  "alpha_pattern": {},
+  "arrow_config": null,
+  "auto_mapping": null,
+  "base_model_name_or_path": "unsloth/llama-3-8b-Instruct-bnb-4bit",
+  "bias": "none",
+  "corda_config": null,
+  "ensure_weight_tying": false,
+  "eva_config": null,
+  "exclude_modules": null,
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 32,
+  "lora_bias": false,
+  "lora_dropout": 0.05,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "peft_version": "0.18.1",
+  "qalora_group_size": 16,
+  "r": 16,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "down_proj",
+    "o_proj",
+    "k_proj",
+    "q_proj",
+    "v_proj",
+    "gate_proj",
+    "up_proj"
+  ],
+  "target_parameters": null,
+  "task_type": "CAUSAL_LM",
+  "trainable_token_indices": null,
+  "use_dora": false,
+  "use_qalora": false,
+  "use_rslora": false
+}

checkpoint-100/adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9a2c2b8092d72e36662bbcb939947cb8b00883bb89d3249cfedbc4e6a800463f
+size 83946192

checkpoint-100/chat_template.jinja

@@ -0,0 +1,5 @@
+{% set loop_messages = messages %}{% for message in loop_messages %}{% set content = '<|start_header_id|>' + message['role'] + '<|end_header_id|>
+
+'+ message['content'] | trim + '<|eot_id|>' %}{% if loop.index0 == 0 %}{% set content = bos_token + content %}{% endif %}{{ content }}{% endfor %}{% if add_generation_prompt %}{{ '<|start_header_id|>assistant<|end_header_id|>
+
+' }}{% endif %}

checkpoint-100/optimizer.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:189d043694baff6a2630a6b1e5a58f07b1e7a255a848278d3f7622fe030eabe4
+size 335817867

checkpoint-100/rng_state.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:656940eec6948746efc59dba0c191ea5ae91cfbd43a4858cae4f839eac52b6a0
+size 14645

checkpoint-100/scheduler.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:eb4fcae01b7d3ddee06d5dff83dd357ce4a03d2635baedaafac11fed9751390f
+size 1465

checkpoint-100/tokenizer.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3c5cf44023714fb39b05e71e425f8d7b92805ff73f7988b083b8c87f0bf87393
+size 17209961

checkpoint-100/tokenizer_config.json

@@ -0,0 +1,15 @@
+{
+  "backend": "tokenizers",
+  "bos_token": "<|begin_of_text|>",
+  "clean_up_tokenization_spaces": true,
+  "eos_token": "<|eot_id|>",
+  "is_local": false,
+  "model_input_names": [
+    "input_ids",
+    "attention_mask"
+  ],
+  "model_max_length": 1000000000000000019884624838656,
+  "pad_token": "<|reserved_special_token_250|>",
+  "padding_side": "left",
+  "tokenizer_class": "TokenizersBackend"
+}

checkpoint-100/trainer_state.json

@@ -0,0 +1,134 @@
+{
+  "best_global_step": null,
+  "best_metric": null,
+  "best_model_checkpoint": null,
+  "epoch": 0.010103561505430665,
+  "eval_steps": 500,
+  "global_step": 100,
+  "is_hyper_param_search": false,
+  "is_local_process_zero": true,
+  "is_world_process_zero": true,
+  "log_history": [
+    {
+      "entropy": 1.45259268283844,
+      "epoch": 0.0010103561505430665,
+      "grad_norm": 0.2451171875,
+      "learning_rate": 0.0001964,
+      "loss": 1.7130912780761718,
+      "mean_token_accuracy": 0.6436435863375664,
+      "num_tokens": 28151.0,
+      "step": 10
+    },
+    {
+      "entropy": 1.3634081721305846,
+      "epoch": 0.002020712301086133,
+      "grad_norm": 0.484375,
+      "learning_rate": 0.00019240000000000001,
+      "loss": 1.3450921058654786,
+      "mean_token_accuracy": 0.686880823969841,
+      "num_tokens": 57374.0,
+      "step": 20
+    },
+    {
+      "entropy": 1.1857430338859558,
+      "epoch": 0.0030310684516291994,
+      "grad_norm": 0.322265625,
+      "learning_rate": 0.0001884,
+      "loss": 1.205996608734131,
+      "mean_token_accuracy": 0.706908255815506,
+      "num_tokens": 86755.0,
+      "step": 30
+    },
+    {
+      "entropy": 1.156875231862068,
+      "epoch": 0.004041424602172266,
+      "grad_norm": 0.359375,
+      "learning_rate": 0.0001844,
+      "loss": 1.1667716026306152,
+      "mean_token_accuracy": 0.7132074117660523,
+      "num_tokens": 114130.0,
+      "step": 40
+    },
+    {
+      "entropy": 1.083250343799591,
+      "epoch": 0.005051780752715332,
+      "grad_norm": 0.330078125,
+      "learning_rate": 0.00018040000000000002,
+      "loss": 1.1047730445861816,
+      "mean_token_accuracy": 0.7196864277124405,
+      "num_tokens": 141768.0,
+      "step": 50
+    },
+    {
+      "entropy": 1.0874410301446915,
+      "epoch": 0.006062136903258399,
+      "grad_norm": 0.265625,
+      "learning_rate": 0.0001764,
+      "loss": 1.0894905090332032,
+      "mean_token_accuracy": 0.7210412979125976,
+      "num_tokens": 172272.0,
+      "step": 60
+    },
+    {
+      "entropy": 1.1303296595811845,
+      "epoch": 0.007072493053801465,
+      "grad_norm": 0.25390625,
+      "learning_rate": 0.00017240000000000002,
+      "loss": 1.1445048332214356,
+      "mean_token_accuracy": 0.7178041815757752,
+      "num_tokens": 200909.0,
+      "step": 70
+    },
+    {
+      "entropy": 1.0938484042882919,
+      "epoch": 0.008082849204344532,
+      "grad_norm": 0.271484375,
+      "learning_rate": 0.0001684,
+      "loss": 1.098367691040039,
+      "mean_token_accuracy": 0.7246084690093995,
+      "num_tokens": 228726.0,
+      "step": 80
+    },
+    {
+      "entropy": 1.0779876083135604,
+      "epoch": 0.009093205354887599,
+      "grad_norm": 0.2216796875,
+      "learning_rate": 0.0001644,
+      "loss": 1.0803230285644532,
+      "mean_token_accuracy": 0.720952507853508,
+      "num_tokens": 255181.0,
+      "step": 90
+    },
+    {
+      "entropy": 1.1645614862442017,
+      "epoch": 0.010103561505430665,
+      "grad_norm": 0.1826171875,
+      "learning_rate": 0.00016040000000000002,
+      "loss": 1.147304630279541,
+      "mean_token_accuracy": 0.7067203193902969,
+      "num_tokens": 283616.0,
+      "step": 100
+    }
+  ],
+  "logging_steps": 10,
+  "max_steps": 500,
+  "num_input_tokens_seen": 0,
+  "num_train_epochs": 1,
+  "save_steps": 100,
+  "stateful_callbacks": {
+    "TrainerControl": {
+      "args": {
+        "should_epoch_stop": false,
+        "should_evaluate": false,
+        "should_log": false,
+        "should_save": true,
+        "should_training_stop": false
+      },
+      "attributes": {}
+    }
+  },
+  "total_flos": 2.9285710718828544e+16,
+  "train_batch_size": 8,
+  "trial_name": null,
+  "trial_params": null
+}

checkpoint-100/training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:421ff9d264ca2f5b99d8b884b2e0d356ce8f07a8e1af063b86e9c6462968a6e6
+size 5649

checkpoint-200/README.md

@@ -0,0 +1,209 @@
+---
+base_model: unsloth/llama-3-8b-Instruct-bnb-4bit
+library_name: peft
+pipeline_tag: text-generation
+tags:
+- base_model:adapter:unsloth/llama-3-8b-Instruct-bnb-4bit
+- lora
+- sft
+- transformers
+- trl
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.18.1

checkpoint-200/adapter_config.json

@@ -0,0 +1,46 @@
+{
+  "alora_invocation_tokens": null,
+  "alpha_pattern": {},
+  "arrow_config": null,
+  "auto_mapping": null,
+  "base_model_name_or_path": "unsloth/llama-3-8b-Instruct-bnb-4bit",
+  "bias": "none",
+  "corda_config": null,
+  "ensure_weight_tying": false,
+  "eva_config": null,
+  "exclude_modules": null,
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 32,
+  "lora_bias": false,
+  "lora_dropout": 0.05,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "peft_version": "0.18.1",
+  "qalora_group_size": 16,
+  "r": 16,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "down_proj",
+    "o_proj",
+    "k_proj",
+    "q_proj",
+    "v_proj",
+    "gate_proj",
+    "up_proj"
+  ],
+  "target_parameters": null,
+  "task_type": "CAUSAL_LM",
+  "trainable_token_indices": null,
+  "use_dora": false,
+  "use_qalora": false,
+  "use_rslora": false
+}

checkpoint-200/adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:09fa87e3d3e86a067d90eaa846ea88c86e8db4d6dfc7c81c48161d222148cc90
+size 83946192

checkpoint-200/chat_template.jinja

@@ -0,0 +1,5 @@
+{% set loop_messages = messages %}{% for message in loop_messages %}{% set content = '<|start_header_id|>' + message['role'] + '<|end_header_id|>
+
+'+ message['content'] | trim + '<|eot_id|>' %}{% if loop.index0 == 0 %}{% set content = bos_token + content %}{% endif %}{{ content }}{% endfor %}{% if add_generation_prompt %}{{ '<|start_header_id|>assistant<|end_header_id|>
+
+' }}{% endif %}

checkpoint-200/optimizer.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3067ac6f15a78cb59a530f24b6f633438906cff514328780a56664d4019d1cac
+size 335817867

checkpoint-200/rng_state.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f83c2fb90a464d2069f8c9696adef67a1221780665f6aa89b1aee6e5e66a9bb1
+size 14645

checkpoint-200/scheduler.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4a0a4d24c23c6486b90f7f10609a9f77a224ecf6d0d34386034a3cba267d7fe8
+size 1465

checkpoint-200/tokenizer.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3c5cf44023714fb39b05e71e425f8d7b92805ff73f7988b083b8c87f0bf87393
+size 17209961

checkpoint-200/tokenizer_config.json

@@ -0,0 +1,15 @@
+{
+  "backend": "tokenizers",
+  "bos_token": "<|begin_of_text|>",
+  "clean_up_tokenization_spaces": true,
+  "eos_token": "<|eot_id|>",
+  "is_local": false,
+  "model_input_names": [
+    "input_ids",
+    "attention_mask"
+  ],
+  "model_max_length": 1000000000000000019884624838656,
+  "pad_token": "<|reserved_special_token_250|>",
+  "padding_side": "left",
+  "tokenizer_class": "TokenizersBackend"
+}

checkpoint-200/trainer_state.json

@@ -0,0 +1,234 @@
+{
+  "best_global_step": null,
+  "best_metric": null,
+  "best_model_checkpoint": null,
+  "epoch": 0.02020712301086133,
+  "eval_steps": 500,
+  "global_step": 200,
+  "is_hyper_param_search": false,
+  "is_local_process_zero": true,
+  "is_world_process_zero": true,
+  "log_history": [
+    {
+      "entropy": 1.45259268283844,
+      "epoch": 0.0010103561505430665,
+      "grad_norm": 0.2451171875,
+      "learning_rate": 0.0001964,
+      "loss": 1.7130912780761718,
+      "mean_token_accuracy": 0.6436435863375664,
+      "num_tokens": 28151.0,
+      "step": 10
+    },
+    {
+      "entropy": 1.3634081721305846,
+      "epoch": 0.002020712301086133,
+      "grad_norm": 0.484375,
+      "learning_rate": 0.00019240000000000001,
+      "loss": 1.3450921058654786,
+      "mean_token_accuracy": 0.686880823969841,
+      "num_tokens": 57374.0,
+      "step": 20
+    },
+    {
+      "entropy": 1.1857430338859558,
+      "epoch": 0.0030310684516291994,
+      "grad_norm": 0.322265625,
+      "learning_rate": 0.0001884,
+      "loss": 1.205996608734131,
+      "mean_token_accuracy": 0.706908255815506,
+      "num_tokens": 86755.0,
+      "step": 30
+    },
+    {
+      "entropy": 1.156875231862068,
+      "epoch": 0.004041424602172266,
+      "grad_norm": 0.359375,
+      "learning_rate": 0.0001844,
+      "loss": 1.1667716026306152,
+      "mean_token_accuracy": 0.7132074117660523,
+      "num_tokens": 114130.0,
+      "step": 40
+    },
+    {
+      "entropy": 1.083250343799591,
+      "epoch": 0.005051780752715332,
+      "grad_norm": 0.330078125,
+      "learning_rate": 0.00018040000000000002,
+      "loss": 1.1047730445861816,
+      "mean_token_accuracy": 0.7196864277124405,
+      "num_tokens": 141768.0,
+      "step": 50
+    },
+    {
+      "entropy": 1.0874410301446915,
+      "epoch": 0.006062136903258399,
+      "grad_norm": 0.265625,
+      "learning_rate": 0.0001764,
+      "loss": 1.0894905090332032,
+      "mean_token_accuracy": 0.7210412979125976,
+      "num_tokens": 172272.0,
+      "step": 60
+    },
+    {
+      "entropy": 1.1303296595811845,
+      "epoch": 0.007072493053801465,
+      "grad_norm": 0.25390625,
+      "learning_rate": 0.00017240000000000002,
+      "loss": 1.1445048332214356,
+      "mean_token_accuracy": 0.7178041815757752,
+      "num_tokens": 200909.0,
+      "step": 70
+    },
+    {
+      "entropy": 1.0938484042882919,
+      "epoch": 0.008082849204344532,
+      "grad_norm": 0.271484375,
+      "learning_rate": 0.0001684,
+      "loss": 1.098367691040039,
+      "mean_token_accuracy": 0.7246084690093995,
+      "num_tokens": 228726.0,
+      "step": 80
+    },
+    {
+      "entropy": 1.0779876083135604,
+      "epoch": 0.009093205354887599,
+      "grad_norm": 0.2216796875,
+      "learning_rate": 0.0001644,
+      "loss": 1.0803230285644532,
+      "mean_token_accuracy": 0.720952507853508,
+      "num_tokens": 255181.0,
+      "step": 90
+    },
+    {
+      "entropy": 1.1645614862442017,
+      "epoch": 0.010103561505430665,
+      "grad_norm": 0.1826171875,
+      "learning_rate": 0.00016040000000000002,
+      "loss": 1.147304630279541,
+      "mean_token_accuracy": 0.7067203193902969,
+      "num_tokens": 283616.0,
+      "step": 100
+    },
+    {
+      "entropy": 1.126008078455925,
+      "epoch": 0.011113917655973731,
+      "grad_norm": 0.2001953125,
+      "learning_rate": 0.0001564,
+      "loss": 1.1415093421936036,
+      "mean_token_accuracy": 0.7101349741220474,
+      "num_tokens": 312151.0,
+      "step": 110
+    },
+    {
+      "entropy": 1.091178685426712,
+      "epoch": 0.012124273806516797,
+      "grad_norm": 0.1953125,
+      "learning_rate": 0.00015240000000000002,
+      "loss": 1.0913947105407715,
+      "mean_token_accuracy": 0.7238801747560502,
+      "num_tokens": 340776.0,
+      "step": 120
+    },
+    {
+      "entropy": 1.2382428109645844,
+      "epoch": 0.013134629957059864,
+      "grad_norm": 0.2099609375,
+      "learning_rate": 0.0001484,
+      "loss": 1.2411503791809082,
+      "mean_token_accuracy": 0.697870621085167,
+      "num_tokens": 371270.0,
+      "step": 130
+    },
+    {
+      "entropy": 1.1168828099966048,
+      "epoch": 0.01414498610760293,
+      "grad_norm": 0.220703125,
+      "learning_rate": 0.0001444,
+      "loss": 1.1341249465942382,
+      "mean_token_accuracy": 0.7141003280878067,
+      "num_tokens": 400176.0,
+      "step": 140
+    },
+    {
+      "entropy": 1.114673560857773,
+      "epoch": 0.015155342258145996,
+      "grad_norm": 0.2109375,
+      "learning_rate": 0.0001404,
+      "loss": 1.1116752624511719,
+      "mean_token_accuracy": 0.7234076589345932,
+      "num_tokens": 427204.0,
+      "step": 150
+    },
+    {
+      "entropy": 1.1378572463989258,
+      "epoch": 0.016165698408689064,
+      "grad_norm": 0.1904296875,
+      "learning_rate": 0.0001364,
+      "loss": 1.1589903831481934,
+      "mean_token_accuracy": 0.7053093910217285,
+      "num_tokens": 458094.0,
+      "step": 160
+    },
+    {
+      "entropy": 1.110730269551277,
+      "epoch": 0.01717605455923213,
+      "grad_norm": 0.1962890625,
+      "learning_rate": 0.00013240000000000002,
+      "loss": 1.087682342529297,
+      "mean_token_accuracy": 0.7177392661571502,
+      "num_tokens": 487098.0,
+      "step": 170
+    },
+    {
+      "entropy": 1.0602406531572341,
+      "epoch": 0.018186410709775197,
+      "grad_norm": 0.228515625,
+      "learning_rate": 0.0001284,
+      "loss": 1.0950936317443847,
+      "mean_token_accuracy": 0.7214239358901977,
+      "num_tokens": 516025.0,
+      "step": 180
+    },
+    {
+      "entropy": 1.1597254037857057,
+      "epoch": 0.01919676686031826,
+      "grad_norm": 0.203125,
+      "learning_rate": 0.00012440000000000002,
+      "loss": 1.1208978652954102,
+      "mean_token_accuracy": 0.7143576145172119,
+      "num_tokens": 544810.0,
+      "step": 190
+    },
+    {
+      "entropy": 1.0519475936889648,
+      "epoch": 0.02020712301086133,
+      "grad_norm": 0.20703125,
+      "learning_rate": 0.0001204,
+      "loss": 1.0744948387145996,
+      "mean_token_accuracy": 0.718455109000206,
+      "num_tokens": 573002.0,
+      "step": 200
+    }
+  ],
+  "logging_steps": 10,
+  "max_steps": 500,
+  "num_input_tokens_seen": 0,
+  "num_train_epochs": 1,
+  "save_steps": 100,
+  "stateful_callbacks": {
+    "TrainerControl": {
+      "args": {
+        "should_epoch_stop": false,
+        "should_evaluate": false,
+        "should_log": false,
+        "should_save": true,
+        "should_training_stop": false
+      },
+      "attributes": {}
+    }
+  },
+  "total_flos": 6.072716910585446e+16,
+  "train_batch_size": 8,
+  "trial_name": null,
+  "trial_params": null
+}

checkpoint-200/training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:421ff9d264ca2f5b99d8b884b2e0d356ce8f07a8e1af063b86e9c6462968a6e6
+size 5649

checkpoint-300/README.md

@@ -0,0 +1,209 @@
+---
+base_model: unsloth/llama-3-8b-Instruct-bnb-4bit
+library_name: peft
+pipeline_tag: text-generation
+tags:
+- base_model:adapter:unsloth/llama-3-8b-Instruct-bnb-4bit
+- lora
+- sft
+- transformers
+- trl
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.18.1

checkpoint-300/adapter_config.json

@@ -0,0 +1,46 @@
+{
+  "alora_invocation_tokens": null,
+  "alpha_pattern": {},
+  "arrow_config": null,
+  "auto_mapping": null,
+  "base_model_name_or_path": "unsloth/llama-3-8b-Instruct-bnb-4bit",
+  "bias": "none",
+  "corda_config": null,
+  "ensure_weight_tying": false,
+  "eva_config": null,
+  "exclude_modules": null,
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 32,
+  "lora_bias": false,
+  "lora_dropout": 0.05,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "peft_version": "0.18.1",
+  "qalora_group_size": 16,
+  "r": 16,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "down_proj",
+    "o_proj",
+    "k_proj",
+    "q_proj",
+    "v_proj",
+    "gate_proj",
+    "up_proj"
+  ],
+  "target_parameters": null,
+  "task_type": "CAUSAL_LM",
+  "trainable_token_indices": null,
+  "use_dora": false,
+  "use_qalora": false,
+  "use_rslora": false
+}

checkpoint-300/adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7cbcf3ca8d5775d18c717b9664d07d0f3dbf4ad047e02f29c45b2aafbe03f792
+size 83946192

checkpoint-300/chat_template.jinja

@@ -0,0 +1,5 @@
+{% set loop_messages = messages %}{% for message in loop_messages %}{% set content = '<|start_header_id|>' + message['role'] + '<|end_header_id|>
+
+'+ message['content'] | trim + '<|eot_id|>' %}{% if loop.index0 == 0 %}{% set content = bos_token + content %}{% endif %}{{ content }}{% endfor %}{% if add_generation_prompt %}{{ '<|start_header_id|>assistant<|end_header_id|>
+
+' }}{% endif %}

checkpoint-300/optimizer.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6fe138a3267a60f56d4593f3a156e8d355dda78fbaf390d33617b908b29bccfb
+size 335818315

checkpoint-300/rng_state.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:298e8a5d47da5232d3f20a30a20c275f9abe3afa14bbb395ac3df2d3ab6f5203
+size 14645

checkpoint-300/scheduler.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ce6e9012b6e51c76f0ec4ef3163722be704ec28c62a3467bda2f81291304af4b
+size 1465

checkpoint-300/tokenizer.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3c5cf44023714fb39b05e71e425f8d7b92805ff73f7988b083b8c87f0bf87393
+size 17209961

checkpoint-300/tokenizer_config.json

@@ -0,0 +1,15 @@
+{
+  "backend": "tokenizers",
+  "bos_token": "<|begin_of_text|>",
+  "clean_up_tokenization_spaces": true,
+  "eos_token": "<|eot_id|>",
+  "is_local": false,
+  "model_input_names": [
+    "input_ids",
+    "attention_mask"
+  ],
+  "model_max_length": 1000000000000000019884624838656,
+  "pad_token": "<|reserved_special_token_250|>",
+  "padding_side": "left",
+  "tokenizer_class": "TokenizersBackend"
+}

checkpoint-300/trainer_state.json

@@ -0,0 +1,334 @@
+{
+  "best_global_step": null,
+  "best_metric": null,
+  "best_model_checkpoint": null,
+  "epoch": 0.030310684516291993,
+  "eval_steps": 500,
+  "global_step": 300,
+  "is_hyper_param_search": false,
+  "is_local_process_zero": true,
+  "is_world_process_zero": true,
+  "log_history": [
+    {
+      "entropy": 1.45259268283844,
+      "epoch": 0.0010103561505430665,
+      "grad_norm": 0.2451171875,
+      "learning_rate": 0.0001964,
+      "loss": 1.7130912780761718,
+      "mean_token_accuracy": 0.6436435863375664,
+      "num_tokens": 28151.0,
+      "step": 10
+    },
+    {
+      "entropy": 1.3634081721305846,
+      "epoch": 0.002020712301086133,
+      "grad_norm": 0.484375,
+      "learning_rate": 0.00019240000000000001,
+      "loss": 1.3450921058654786,
+      "mean_token_accuracy": 0.686880823969841,
+      "num_tokens": 57374.0,
+      "step": 20
+    },
+    {
+      "entropy": 1.1857430338859558,
+      "epoch": 0.0030310684516291994,
+      "grad_norm": 0.322265625,
+      "learning_rate": 0.0001884,
+      "loss": 1.205996608734131,
+      "mean_token_accuracy": 0.706908255815506,
+      "num_tokens": 86755.0,
+      "step": 30
+    },
+    {
+      "entropy": 1.156875231862068,
+      "epoch": 0.004041424602172266,
+      "grad_norm": 0.359375,
+      "learning_rate": 0.0001844,
+      "loss": 1.1667716026306152,
+      "mean_token_accuracy": 0.7132074117660523,
+      "num_tokens": 114130.0,
+      "step": 40
+    },
+    {
+      "entropy": 1.083250343799591,
+      "epoch": 0.005051780752715332,
+      "grad_norm": 0.330078125,
+      "learning_rate": 0.00018040000000000002,
+      "loss": 1.1047730445861816,
+      "mean_token_accuracy": 0.7196864277124405,
+      "num_tokens": 141768.0,
+      "step": 50
+    },
+    {
+      "entropy": 1.0874410301446915,
+      "epoch": 0.006062136903258399,
+      "grad_norm": 0.265625,
+      "learning_rate": 0.0001764,
+      "loss": 1.0894905090332032,
+      "mean_token_accuracy": 0.7210412979125976,
+      "num_tokens": 172272.0,
+      "step": 60
+    },
+    {
+      "entropy": 1.1303296595811845,
+      "epoch": 0.007072493053801465,
+      "grad_norm": 0.25390625,
+      "learning_rate": 0.00017240000000000002,
+      "loss": 1.1445048332214356,
+      "mean_token_accuracy": 0.7178041815757752,
+      "num_tokens": 200909.0,
+      "step": 70
+    },
+    {
+      "entropy": 1.0938484042882919,
+      "epoch": 0.008082849204344532,
+      "grad_norm": 0.271484375,
+      "learning_rate": 0.0001684,
+      "loss": 1.098367691040039,
+      "mean_token_accuracy": 0.7246084690093995,
+      "num_tokens": 228726.0,
+      "step": 80
+    },
+    {
+      "entropy": 1.0779876083135604,
+      "epoch": 0.009093205354887599,
+      "grad_norm": 0.2216796875,
+      "learning_rate": 0.0001644,
+      "loss": 1.0803230285644532,
+      "mean_token_accuracy": 0.720952507853508,
+      "num_tokens": 255181.0,
+      "step": 90
+    },
+    {
+      "entropy": 1.1645614862442017,
+      "epoch": 0.010103561505430665,
+      "grad_norm": 0.1826171875,
+      "learning_rate": 0.00016040000000000002,
+      "loss": 1.147304630279541,
+      "mean_token_accuracy": 0.7067203193902969,
+      "num_tokens": 283616.0,
+      "step": 100
+    },
+    {
+      "entropy": 1.126008078455925,
+      "epoch": 0.011113917655973731,
+      "grad_norm": 0.2001953125,
+      "learning_rate": 0.0001564,
+      "loss": 1.1415093421936036,
+      "mean_token_accuracy": 0.7101349741220474,
+      "num_tokens": 312151.0,
+      "step": 110
+    },
+    {
+      "entropy": 1.091178685426712,
+      "epoch": 0.012124273806516797,
+      "grad_norm": 0.1953125,
+      "learning_rate": 0.00015240000000000002,
+      "loss": 1.0913947105407715,
+      "mean_token_accuracy": 0.7238801747560502,
+      "num_tokens": 340776.0,
+      "step": 120
+    },
+    {
+      "entropy": 1.2382428109645844,
+      "epoch": 0.013134629957059864,
+      "grad_norm": 0.2099609375,
+      "learning_rate": 0.0001484,
+      "loss": 1.2411503791809082,
+      "mean_token_accuracy": 0.697870621085167,
+      "num_tokens": 371270.0,
+      "step": 130
+    },
+    {
+      "entropy": 1.1168828099966048,
+      "epoch": 0.01414498610760293,
+      "grad_norm": 0.220703125,
+      "learning_rate": 0.0001444,
+      "loss": 1.1341249465942382,
+      "mean_token_accuracy": 0.7141003280878067,
+      "num_tokens": 400176.0,
+      "step": 140
+    },
+    {
+      "entropy": 1.114673560857773,
+      "epoch": 0.015155342258145996,
+      "grad_norm": 0.2109375,
+      "learning_rate": 0.0001404,
+      "loss": 1.1116752624511719,
+      "mean_token_accuracy": 0.7234076589345932,
+      "num_tokens": 427204.0,
+      "step": 150
+    },
+    {
+      "entropy": 1.1378572463989258,
+      "epoch": 0.016165698408689064,
+      "grad_norm": 0.1904296875,
+      "learning_rate": 0.0001364,
+      "loss": 1.1589903831481934,
+      "mean_token_accuracy": 0.7053093910217285,
+      "num_tokens": 458094.0,
+      "step": 160
+    },
+    {
+      "entropy": 1.110730269551277,
+      "epoch": 0.01717605455923213,
+      "grad_norm": 0.1962890625,
+      "learning_rate": 0.00013240000000000002,
+      "loss": 1.087682342529297,
+      "mean_token_accuracy": 0.7177392661571502,
+      "num_tokens": 487098.0,
+      "step": 170
+    },
+    {
+      "entropy": 1.0602406531572341,
+      "epoch": 0.018186410709775197,
+      "grad_norm": 0.228515625,
+      "learning_rate": 0.0001284,
+      "loss": 1.0950936317443847,
+      "mean_token_accuracy": 0.7214239358901977,
+      "num_tokens": 516025.0,
+      "step": 180
+    },
+    {
+      "entropy": 1.1597254037857057,
+      "epoch": 0.01919676686031826,
+      "grad_norm": 0.203125,
+      "learning_rate": 0.00012440000000000002,
+      "loss": 1.1208978652954102,
+      "mean_token_accuracy": 0.7143576145172119,
+      "num_tokens": 544810.0,
+      "step": 190
+    },
+    {
+      "entropy": 1.0519475936889648,
+      "epoch": 0.02020712301086133,
+      "grad_norm": 0.20703125,
+      "learning_rate": 0.0001204,
+      "loss": 1.0744948387145996,
+      "mean_token_accuracy": 0.718455109000206,
+      "num_tokens": 573002.0,
+      "step": 200
+    },
+    {
+      "entropy": 1.2084601551294327,
+      "epoch": 0.021217479161404394,
+      "grad_norm": 0.1943359375,
+      "learning_rate": 0.0001164,
+      "loss": 1.2174930572509766,
+      "mean_token_accuracy": 0.6999662011861801,
+      "num_tokens": 602401.0,
+      "step": 210
+    },
+    {
+      "entropy": 1.1912338614463807,
+      "epoch": 0.022227835311947462,
+      "grad_norm": 0.216796875,
+      "learning_rate": 0.00011240000000000002,
+      "loss": 1.183759880065918,
+      "mean_token_accuracy": 0.7107820093631745,
+      "num_tokens": 629994.0,
+      "step": 220
+    },
+    {
+      "entropy": 1.0905429303646088,
+      "epoch": 0.023238191462490527,
+      "grad_norm": 0.203125,
+      "learning_rate": 0.00010840000000000002,
+      "loss": 1.086796474456787,
+      "mean_token_accuracy": 0.7176417618989944,
+      "num_tokens": 658686.0,
+      "step": 230
+    },
+    {
+      "entropy": 1.0157978028059005,
+      "epoch": 0.024248547613033595,
+      "grad_norm": 0.23828125,
+      "learning_rate": 0.0001044,
+      "loss": 1.012559700012207,
+      "mean_token_accuracy": 0.7369582027196884,
+      "num_tokens": 685539.0,
+      "step": 240
+    },
+    {
+      "entropy": 1.1027084678411483,
+      "epoch": 0.02525890376357666,
+      "grad_norm": 0.2314453125,
+      "learning_rate": 0.0001004,
+      "loss": 1.1228812217712403,
+      "mean_token_accuracy": 0.7192230314016342,
+      "num_tokens": 715224.0,
+      "step": 250
+    },
+    {
+      "entropy": 1.0666967660188675,
+      "epoch": 0.026269259914119727,
+      "grad_norm": 0.2412109375,
+      "learning_rate": 9.64e-05,
+      "loss": 1.0753504753112793,
+      "mean_token_accuracy": 0.719659861922264,
+      "num_tokens": 745782.0,
+      "step": 260
+    },
+    {
+      "entropy": 1.034983891248703,
+      "epoch": 0.027279616064662792,
+      "grad_norm": 0.2216796875,
+      "learning_rate": 9.240000000000001e-05,
+      "loss": 1.032216453552246,
+      "mean_token_accuracy": 0.7364529073238373,
+      "num_tokens": 772358.0,
+      "step": 270
+    },
+    {
+      "entropy": 1.0821890532970428,
+      "epoch": 0.02828997221520586,
+      "grad_norm": 0.2294921875,
+      "learning_rate": 8.840000000000001e-05,
+      "loss": 1.0743555068969726,
+      "mean_token_accuracy": 0.7248906105756759,
+      "num_tokens": 800173.0,
+      "step": 280
+    },
+    {
+      "entropy": 1.1281798005104064,
+      "epoch": 0.029300328365748928,
+      "grad_norm": 0.2216796875,
+      "learning_rate": 8.44e-05,
+      "loss": 1.1471177101135255,
+      "mean_token_accuracy": 0.7095231086015701,
+      "num_tokens": 826835.0,
+      "step": 290
+    },
+    {
+      "entropy": 1.0436414241790772,
+      "epoch": 0.030310684516291993,
+      "grad_norm": 0.2001953125,
+      "learning_rate": 8.04e-05,
+      "loss": 1.0298893928527832,
+      "mean_token_accuracy": 0.7325254619121552,
+      "num_tokens": 853598.0,
+      "step": 300
+    }
+  ],
+  "logging_steps": 10,
+  "max_steps": 500,
+  "num_input_tokens_seen": 0,
+  "num_train_epochs": 1,
+  "save_steps": 100,
+  "stateful_callbacks": {
+    "TrainerControl": {
+      "args": {
+        "should_epoch_stop": false,
+        "should_evaluate": false,
+        "should_log": false,
+        "should_save": true,
+        "should_training_stop": false
+      },
+      "attributes": {}
+    }
+  },
+  "total_flos": 9.026935257700762e+16,
+  "train_batch_size": 8,
+  "trial_name": null,
+  "trial_params": null
+}

checkpoint-300/training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:421ff9d264ca2f5b99d8b884b2e0d356ce8f07a8e1af063b86e9c6462968a6e6
+size 5649

checkpoint-400/README.md

@@ -0,0 +1,209 @@
+---
+base_model: unsloth/llama-3-8b-Instruct-bnb-4bit
+library_name: peft
+pipeline_tag: text-generation
+tags:
+- base_model:adapter:unsloth/llama-3-8b-Instruct-bnb-4bit
+- lora
+- sft
+- transformers
+- trl
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.18.1

checkpoint-400/adapter_config.json

@@ -0,0 +1,46 @@
+{
+  "alora_invocation_tokens": null,
+  "alpha_pattern": {},
+  "arrow_config": null,
+  "auto_mapping": null,
+  "base_model_name_or_path": "unsloth/llama-3-8b-Instruct-bnb-4bit",
+  "bias": "none",
+  "corda_config": null,
+  "ensure_weight_tying": false,
+  "eva_config": null,
+  "exclude_modules": null,
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 32,
+  "lora_bias": false,
+  "lora_dropout": 0.05,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "peft_version": "0.18.1",
+  "qalora_group_size": 16,
+  "r": 16,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "down_proj",
+    "o_proj",
+    "k_proj",
+    "q_proj",
+    "v_proj",
+    "gate_proj",
+    "up_proj"
+  ],
+  "target_parameters": null,
+  "task_type": "CAUSAL_LM",
+  "trainable_token_indices": null,
+  "use_dora": false,
+  "use_qalora": false,
+  "use_rslora": false
+}

checkpoint-400/adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ed830c28dd9009fd9f8f9ef49fadbe801b9643637319560ff6802f35368f57f7
+size 83946192

checkpoint-400/chat_template.jinja

@@ -0,0 +1,5 @@
+{% set loop_messages = messages %}{% for message in loop_messages %}{% set content = '<|start_header_id|>' + message['role'] + '<|end_header_id|>
+
+'+ message['content'] | trim + '<|eot_id|>' %}{% if loop.index0 == 0 %}{% set content = bos_token + content %}{% endif %}{{ content }}{% endfor %}{% if add_generation_prompt %}{{ '<|start_header_id|>assistant<|end_header_id|>
+
+' }}{% endif %}

checkpoint-400/optimizer.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a29a5e86755e5f4d7bf3569dfbccdee1d4b290afa30db3b8c00266514c2b8248
+size 335818315

checkpoint-400/rng_state.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:edf3e1d34f77115ba655f936fb1927d096562299f766f37a65033d66f88d36c4
+size 14645

checkpoint-400/scheduler.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a488d0df71c0c4ad51b6aee82ccb2de3b999cb205d5e4fd378a76bd6378e758b
+size 1465

checkpoint-400/tokenizer.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3c5cf44023714fb39b05e71e425f8d7b92805ff73f7988b083b8c87f0bf87393
+size 17209961

checkpoint-400/tokenizer_config.json

@@ -0,0 +1,15 @@
+{
+  "backend": "tokenizers",
+  "bos_token": "<|begin_of_text|>",
+  "clean_up_tokenization_spaces": true,
+  "eos_token": "<|eot_id|>",
+  "is_local": false,
+  "model_input_names": [
+    "input_ids",
+    "attention_mask"
+  ],
+  "model_max_length": 1000000000000000019884624838656,
+  "pad_token": "<|reserved_special_token_250|>",
+  "padding_side": "left",
+  "tokenizer_class": "TokenizersBackend"
+}

checkpoint-400/trainer_state.json

@@ -0,0 +1,434 @@
+{
+  "best_global_step": null,
+  "best_metric": null,
+  "best_model_checkpoint": null,
+  "epoch": 0.04041424602172266,
+  "eval_steps": 500,
+  "global_step": 400,
+  "is_hyper_param_search": false,
+  "is_local_process_zero": true,
+  "is_world_process_zero": true,
+  "log_history": [
+    {
+      "entropy": 1.45259268283844,
+      "epoch": 0.0010103561505430665,
+      "grad_norm": 0.2451171875,
+      "learning_rate": 0.0001964,
+      "loss": 1.7130912780761718,
+      "mean_token_accuracy": 0.6436435863375664,
+      "num_tokens": 28151.0,
+      "step": 10
+    },
+    {
+      "entropy": 1.3634081721305846,
+      "epoch": 0.002020712301086133,
+      "grad_norm": 0.484375,
+      "learning_rate": 0.00019240000000000001,
+      "loss": 1.3450921058654786,
+      "mean_token_accuracy": 0.686880823969841,
+      "num_tokens": 57374.0,
+      "step": 20
+    },
+    {
+      "entropy": 1.1857430338859558,
+      "epoch": 0.0030310684516291994,
+      "grad_norm": 0.322265625,
+      "learning_rate": 0.0001884,
+      "loss": 1.205996608734131,
+      "mean_token_accuracy": 0.706908255815506,
+      "num_tokens": 86755.0,
+      "step": 30
+    },
+    {
+      "entropy": 1.156875231862068,
+      "epoch": 0.004041424602172266,
+      "grad_norm": 0.359375,
+      "learning_rate": 0.0001844,
+      "loss": 1.1667716026306152,
+      "mean_token_accuracy": 0.7132074117660523,
+      "num_tokens": 114130.0,
+      "step": 40
+    },
+    {
+      "entropy": 1.083250343799591,
+      "epoch": 0.005051780752715332,
+      "grad_norm": 0.330078125,
+      "learning_rate": 0.00018040000000000002,
+      "loss": 1.1047730445861816,
+      "mean_token_accuracy": 0.7196864277124405,
+      "num_tokens": 141768.0,
+      "step": 50
+    },
+    {
+      "entropy": 1.0874410301446915,
+      "epoch": 0.006062136903258399,
+      "grad_norm": 0.265625,
+      "learning_rate": 0.0001764,
+      "loss": 1.0894905090332032,
+      "mean_token_accuracy": 0.7210412979125976,
+      "num_tokens": 172272.0,
+      "step": 60
+    },
+    {
+      "entropy": 1.1303296595811845,
+      "epoch": 0.007072493053801465,
+      "grad_norm": 0.25390625,
+      "learning_rate": 0.00017240000000000002,
+      "loss": 1.1445048332214356,
+      "mean_token_accuracy": 0.7178041815757752,
+      "num_tokens": 200909.0,
+      "step": 70
+    },
+    {
+      "entropy": 1.0938484042882919,
+      "epoch": 0.008082849204344532,
+      "grad_norm": 0.271484375,
+      "learning_rate": 0.0001684,
+      "loss": 1.098367691040039,
+      "mean_token_accuracy": 0.7246084690093995,
+      "num_tokens": 228726.0,
+      "step": 80
+    },
+    {
+      "entropy": 1.0779876083135604,
+      "epoch": 0.009093205354887599,
+      "grad_norm": 0.2216796875,
+      "learning_rate": 0.0001644,
+      "loss": 1.0803230285644532,
+      "mean_token_accuracy": 0.720952507853508,
+      "num_tokens": 255181.0,
+      "step": 90
+    },
+    {
+      "entropy": 1.1645614862442017,
+      "epoch": 0.010103561505430665,
+      "grad_norm": 0.1826171875,
+      "learning_rate": 0.00016040000000000002,
+      "loss": 1.147304630279541,
+      "mean_token_accuracy": 0.7067203193902969,
+      "num_tokens": 283616.0,
+      "step": 100
+    },
+    {
+      "entropy": 1.126008078455925,
+      "epoch": 0.011113917655973731,
+      "grad_norm": 0.2001953125,
+      "learning_rate": 0.0001564,
+      "loss": 1.1415093421936036,
+      "mean_token_accuracy": 0.7101349741220474,
+      "num_tokens": 312151.0,
+      "step": 110
+    },
+    {
+      "entropy": 1.091178685426712,
+      "epoch": 0.012124273806516797,
+      "grad_norm": 0.1953125,
+      "learning_rate": 0.00015240000000000002,
+      "loss": 1.0913947105407715,
+      "mean_token_accuracy": 0.7238801747560502,
+      "num_tokens": 340776.0,
+      "step": 120
+    },
+    {
+      "entropy": 1.2382428109645844,
+      "epoch": 0.013134629957059864,
+      "grad_norm": 0.2099609375,
+      "learning_rate": 0.0001484,
+      "loss": 1.2411503791809082,
+      "mean_token_accuracy": 0.697870621085167,
+      "num_tokens": 371270.0,
+      "step": 130
+    },
+    {
+      "entropy": 1.1168828099966048,
+      "epoch": 0.01414498610760293,
+      "grad_norm": 0.220703125,
+      "learning_rate": 0.0001444,
+      "loss": 1.1341249465942382,
+      "mean_token_accuracy": 0.7141003280878067,
+      "num_tokens": 400176.0,
+      "step": 140
+    },
+    {
+      "entropy": 1.114673560857773,
+      "epoch": 0.015155342258145996,
+      "grad_norm": 0.2109375,
+      "learning_rate": 0.0001404,
+      "loss": 1.1116752624511719,
+      "mean_token_accuracy": 0.7234076589345932,
+      "num_tokens": 427204.0,
+      "step": 150
+    },
+    {
+      "entropy": 1.1378572463989258,
+      "epoch": 0.016165698408689064,
+      "grad_norm": 0.1904296875,
+      "learning_rate": 0.0001364,
+      "loss": 1.1589903831481934,
+      "mean_token_accuracy": 0.7053093910217285,
+      "num_tokens": 458094.0,
+      "step": 160
+    },
+    {
+      "entropy": 1.110730269551277,
+      "epoch": 0.01717605455923213,
+      "grad_norm": 0.1962890625,
+      "learning_rate": 0.00013240000000000002,
+      "loss": 1.087682342529297,
+      "mean_token_accuracy": 0.7177392661571502,
+      "num_tokens": 487098.0,
+      "step": 170
+    },
+    {
+      "entropy": 1.0602406531572341,
+      "epoch": 0.018186410709775197,
+      "grad_norm": 0.228515625,
+      "learning_rate": 0.0001284,
+      "loss": 1.0950936317443847,
+      "mean_token_accuracy": 0.7214239358901977,
+      "num_tokens": 516025.0,
+      "step": 180
+    },
+    {
+      "entropy": 1.1597254037857057,
+      "epoch": 0.01919676686031826,
+      "grad_norm": 0.203125,
+      "learning_rate": 0.00012440000000000002,
+      "loss": 1.1208978652954102,
+      "mean_token_accuracy": 0.7143576145172119,
+      "num_tokens": 544810.0,
+      "step": 190
+    },
+    {
+      "entropy": 1.0519475936889648,
+      "epoch": 0.02020712301086133,
+      "grad_norm": 0.20703125,
+      "learning_rate": 0.0001204,
+      "loss": 1.0744948387145996,
+      "mean_token_accuracy": 0.718455109000206,
+      "num_tokens": 573002.0,
+      "step": 200
+    },
+    {
+      "entropy": 1.2084601551294327,
+      "epoch": 0.021217479161404394,
+      "grad_norm": 0.1943359375,
+      "learning_rate": 0.0001164,
+      "loss": 1.2174930572509766,
+      "mean_token_accuracy": 0.6999662011861801,
+      "num_tokens": 602401.0,
+      "step": 210
+    },
+    {
+      "entropy": 1.1912338614463807,
+      "epoch": 0.022227835311947462,
+      "grad_norm": 0.216796875,
+      "learning_rate": 0.00011240000000000002,
+      "loss": 1.183759880065918,
+      "mean_token_accuracy": 0.7107820093631745,
+      "num_tokens": 629994.0,
+      "step": 220
+    },
+    {
+      "entropy": 1.0905429303646088,
+      "epoch": 0.023238191462490527,
+      "grad_norm": 0.203125,
+      "learning_rate": 0.00010840000000000002,
+      "loss": 1.086796474456787,
+      "mean_token_accuracy": 0.7176417618989944,
+      "num_tokens": 658686.0,
+      "step": 230
+    },
+    {
+      "entropy": 1.0157978028059005,
+      "epoch": 0.024248547613033595,
+      "grad_norm": 0.23828125,
+      "learning_rate": 0.0001044,
+      "loss": 1.012559700012207,
+      "mean_token_accuracy": 0.7369582027196884,
+      "num_tokens": 685539.0,
+      "step": 240
+    },
+    {
+      "entropy": 1.1027084678411483,
+      "epoch": 0.02525890376357666,
+      "grad_norm": 0.2314453125,
+      "learning_rate": 0.0001004,
+      "loss": 1.1228812217712403,
+      "mean_token_accuracy": 0.7192230314016342,
+      "num_tokens": 715224.0,
+      "step": 250
+    },
+    {
+      "entropy": 1.0666967660188675,
+      "epoch": 0.026269259914119727,
+      "grad_norm": 0.2412109375,
+      "learning_rate": 9.64e-05,
+      "loss": 1.0753504753112793,
+      "mean_token_accuracy": 0.719659861922264,
+      "num_tokens": 745782.0,
+      "step": 260
+    },
+    {
+      "entropy": 1.034983891248703,
+      "epoch": 0.027279616064662792,
+      "grad_norm": 0.2216796875,
+      "learning_rate": 9.240000000000001e-05,
+      "loss": 1.032216453552246,
+      "mean_token_accuracy": 0.7364529073238373,
+      "num_tokens": 772358.0,
+      "step": 270
+    },
+    {
+      "entropy": 1.0821890532970428,
+      "epoch": 0.02828997221520586,
+      "grad_norm": 0.2294921875,
+      "learning_rate": 8.840000000000001e-05,
+      "loss": 1.0743555068969726,
+      "mean_token_accuracy": 0.7248906105756759,
+      "num_tokens": 800173.0,
+      "step": 280
+    },
+    {
+      "entropy": 1.1281798005104064,
+      "epoch": 0.029300328365748928,
+      "grad_norm": 0.2216796875,
+      "learning_rate": 8.44e-05,
+      "loss": 1.1471177101135255,
+      "mean_token_accuracy": 0.7095231086015701,
+      "num_tokens": 826835.0,
+      "step": 290
+    },
+    {
+      "entropy": 1.0436414241790772,
+      "epoch": 0.030310684516291993,
+      "grad_norm": 0.2001953125,
+      "learning_rate": 8.04e-05,
+      "loss": 1.0298893928527832,
+      "mean_token_accuracy": 0.7325254619121552,
+      "num_tokens": 853598.0,
+      "step": 300
+    },
+    {
+      "entropy": 1.0947031527757645,
+      "epoch": 0.03132104066683506,
+      "grad_norm": 0.232421875,
+      "learning_rate": 7.64e-05,
+      "loss": 1.0945837020874023,
+      "mean_token_accuracy": 0.7231394708156585,
+      "num_tokens": 884023.0,
+      "step": 310
+    },
+    {
+      "entropy": 1.1012510120868684,
+      "epoch": 0.03233139681737813,
+      "grad_norm": 0.17578125,
+      "learning_rate": 7.24e-05,
+      "loss": 1.106449031829834,
+      "mean_token_accuracy": 0.7150771111249924,
+      "num_tokens": 910709.0,
+      "step": 320
+    },
+    {
+      "entropy": 1.0871451586484908,
+      "epoch": 0.03334175296792119,
+      "grad_norm": 0.2041015625,
+      "learning_rate": 6.840000000000001e-05,
+      "loss": 1.0799496650695801,
+      "mean_token_accuracy": 0.7183821439743042,
+      "num_tokens": 938961.0,
+      "step": 330
+    },
+    {
+      "entropy": 1.0514528155326843,
+      "epoch": 0.03435210911846426,
+      "grad_norm": 0.2080078125,
+      "learning_rate": 6.440000000000001e-05,
+      "loss": 1.081194019317627,
+      "mean_token_accuracy": 0.7243827939033508,
+      "num_tokens": 967584.0,
+      "step": 340
+    },
+    {
+      "entropy": 1.1092546790838242,
+      "epoch": 0.035362465269007326,
+      "grad_norm": 0.201171875,
+      "learning_rate": 6.04e-05,
+      "loss": 1.0813716888427733,
+      "mean_token_accuracy": 0.7228824734687805,
+      "num_tokens": 995895.0,
+      "step": 350
+    },
+    {
+      "entropy": 1.0287963569164276,
+      "epoch": 0.036372821419550394,
+      "grad_norm": 0.265625,
+      "learning_rate": 5.6399999999999995e-05,
+      "loss": 1.0419590950012207,
+      "mean_token_accuracy": 0.7281966924667358,
+      "num_tokens": 1024358.0,
+      "step": 360
+    },
+    {
+      "entropy": 1.121953997015953,
+      "epoch": 0.037383177570093455,
+      "grad_norm": 0.26171875,
+      "learning_rate": 5.2400000000000007e-05,
+      "loss": 1.0987840652465821,
+      "mean_token_accuracy": 0.7203631848096848,
+      "num_tokens": 1052323.0,
+      "step": 370
+    },
+    {
+      "entropy": 1.110439071059227,
+      "epoch": 0.03839353372063652,
+      "grad_norm": 0.291015625,
+      "learning_rate": 4.8400000000000004e-05,
+      "loss": 1.0895070075988769,
+      "mean_token_accuracy": 0.7241304695606232,
+      "num_tokens": 1079594.0,
+      "step": 380
+    },
+    {
+      "entropy": 1.070469456911087,
+      "epoch": 0.03940388987117959,
+      "grad_norm": 0.3046875,
+      "learning_rate": 4.44e-05,
+      "loss": 1.1078936576843261,
+      "mean_token_accuracy": 0.7139627873897553,
+      "num_tokens": 1110202.0,
+      "step": 390
+    },
+    {
+      "entropy": 1.1429108887910844,
+      "epoch": 0.04041424602172266,
+      "grad_norm": 0.265625,
+      "learning_rate": 4.0400000000000006e-05,
+      "loss": 1.1321399688720704,
+      "mean_token_accuracy": 0.7121616303920746,
+      "num_tokens": 1138357.0,
+      "step": 400
+    }
+  ],
+  "logging_steps": 10,
+  "max_steps": 500,
+  "num_input_tokens_seen": 0,
+  "num_train_epochs": 1,
+  "save_steps": 100,
+  "stateful_callbacks": {
+    "TrainerControl": {
+      "args": {
+        "should_epoch_stop": false,
+        "should_evaluate": false,
+        "should_log": false,
+        "should_save": true,
+        "should_training_stop": false
+      },
+      "attributes": {}
+    }
+  },
+  "total_flos": 1.1975538735238349e+17,
+  "train_batch_size": 8,
+  "trial_name": null,
+  "trial_params": null
+}

checkpoint-400/training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:421ff9d264ca2f5b99d8b884b2e0d356ce8f07a8e1af063b86e9c6462968a6e6
+size 5649

checkpoint-500/README.md

@@ -0,0 +1,209 @@
+---
+base_model: unsloth/llama-3-8b-Instruct-bnb-4bit
+library_name: peft
+pipeline_tag: text-generation
+tags:
+- base_model:adapter:unsloth/llama-3-8b-Instruct-bnb-4bit
+- lora
+- sft
+- transformers
+- trl
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.18.1