End of training

README.md

@@ -1,10 +1,12 @@
 ---
 library_name: transformers
-license: llama3
-base_model: meta-llama/Meta-Llama-3-8B
+license: llama3.1
+base_model: meta-llama/Llama-3.1-8B-Instruct
 tags:
 - axolotl
 - generated_from_trainer
+datasets:
+- Sandevistan_cleaned.jsonl
 model-index:
 - name: L3-Pneuma-8B
   results: []
@@ -16,9 +18,9 @@ should probably proofread and complete it, then remove this comment. -->
 [<img src="https://raw.githubusercontent.com/axolotl-ai-cloud/axolotl/main/image/axolotl-badge-web.png" alt="Built with Axolotl" width="200" height="32"/>](https://github.com/axolotl-ai-cloud/axolotl)
 <details><summary>See axolotl config</summary>
 
-axolotl version: `0.4.1`
+axolotl version: `0.8.0`
 ```yaml
-base_model: meta-llama/Meta-Llama-3-8B
+base_model: meta-llama/Llama-3.1-8B-Instruct
 
 load_in_8bit: false
 load_in_4bit: false
@@ -29,12 +31,11 @@ load_in_4bit: false
 strict: false
 
 datasets:
-  - path: Kquant03/Sandevistan_Reformat
+  - path: Sandevistan_cleaned.jsonl
     type: customllama3_stan
 dataset_prepared_path: last_run_prepared
 val_set_size: 0.05
 output_dir: ./outputs/out
-max_steps: 80000
 
 fix_untrained_tokens: true
 
@@ -50,10 +51,10 @@ wandb_log_model:
 
 gradient_accumulation_steps: 16
 micro_batch_size: 8
-num_epochs: 1
+num_epochs: 2
 optimizer: paged_adamw_8bit
 lr_scheduler: cosine
-learning_rate: 0.00001
+learning_rate: 0.000075
 max_grad_norm: 1
 
 train_on_inputs: false
@@ -94,54 +95,61 @@ special_tokens:
   eos_token: "<|end_of_text|>"
   pad_token: "<|end_of_text|>"
 tokens:
+
 ```
 
 </details><br>
 
 # L3-Pneuma-8B
 
-This model is a fine-tuned version of [meta-llama/Meta-Llama-3-8B](https://huggingface.co/meta-llama/Meta-Llama-3-8B) on the [Sandevistan](https://huggingface.co/datasets/Replete-AI/Sandevistan) dataset.
+This model is a fine-tuned version of [meta-llama/Llama-3.1-8B-Instruct](https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct) on the Sandevistan_cleaned.jsonl dataset.
 It achieves the following results on the evaluation set:
-- Loss: 2.7381
+- Loss: 0.7796
 
 ## Model description
 
-This model is designed to challenge common paradigms in training Large Language Models, giving them a focus on user experience over profitability. These are highly experimental, and need preference training in order to increase their effectiveness.
+More information needed
 
 ## Intended uses & limitations
 
-Chatting, conversation, and assistance in small downstream tasks.
+More information needed
+
+## Training and evaluation data
 
-Large Language Models work incredibly differently from humans, so while we are capable of training and rewarding them to act just like us in many ways, you should treat it as a simulation and use the Socratic method when engaging with them. You, as an end-user should always remain in control of your own thoughts and decisions, and use AI as a way to improve yourself rather than becoming dependent on it.
+More information needed
 
 ## Training procedure
 
 ### Training hyperparameters
 
 The following hyperparameters were used during training:
-- learning_rate: 1e-05
+- learning_rate: 7.5e-05
 - train_batch_size: 8
 - eval_batch_size: 8
 - seed: 42
 - gradient_accumulation_steps: 16
 - total_train_batch_size: 128
-- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
+- optimizer: Use OptimizerNames.PAGED_ADAMW_8BIT with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
 - lr_scheduler_type: cosine
 - lr_scheduler_warmup_steps: 10
-- training_steps: 743
+- num_epochs: 2.0
 
 ### Training results
 
 | Training Loss | Epoch  | Step | Validation Loss |
 |:-------------:|:------:|:----:|:---------------:|
-| 1.0378        | 0.0013 | 1    | 3.0437          |
-| 0.6816        | 0.3334 | 248  | 2.7341          |
-| 0.6543        | 0.6667 | 496  | 2.7381          |
+| 1.3399        | 0.0023 | 1    | 1.3175          |
+| 0.846         | 0.3332 | 143  | 0.8312          |
+| 0.8103        | 0.6665 | 286  | 0.8021          |
+| 0.7617        | 0.9997 | 429  | 0.7737          |
+| 0.5824        | 1.3309 | 572  | 0.7851          |
+| 0.5651        | 1.6641 | 715  | 0.7798          |
+| 0.5738        | 1.9974 | 858  | 0.7796          |
 
 
 ### Framework versions
 
-- Transformers 4.45.1
-- Pytorch 2.3.1+cu121
-- Datasets 2.21.0
-- Tokenizers 0.20.1
+- Transformers 4.51.3
+- Pytorch 2.6.0+cu124
+- Datasets 3.5.0
+- Tokenizers 0.21.1