End of training

README.md

@@ -1,7 +1,7 @@
 ---
 library_name: transformers
-license: other
-base_model: facebook/opt-125m
+license: apache-2.0
+base_model: TinyLlama/TinyLlama-1.1B-Chat-v1.0
 tags:
 - axolotl
 - generated_from_trainer
@@ -20,8 +20,8 @@ should probably proofread and complete it, then remove this comment. -->
 
 axolotl version: `0.6.0`
 ```yaml
-base_model: facebook/opt-125m
-batch_size: 256
+base_model: TinyLlama/TinyLlama-1.1B-Chat-v1.0
+batch_size: 96
 bf16: true
 chat_template: tokenizer_default_fallback_alpaca
 datasets:
@@ -37,7 +37,7 @@ datasets:
     system_prompt: ''
 device_map: auto
 eval_sample_packing: false
-eval_steps: 40
+eval_steps: 200
 flash_attention: true
 gradient_checkpointing: true
 group_by_length: true
@@ -47,7 +47,7 @@ learning_rate: 0.0002
 logging_steps: 10
 lr_scheduler: cosine
 max_steps: 20000
-micro_batch_size: 32
+micro_batch_size: 12
 model_type: AutoModelForCausalLM
 num_epochs: 100
 optimizer: adamw_bnb_8bit
@@ -55,16 +55,16 @@ output_dir: /root/.sn56/axolotl/tmp/test-repo
 pad_to_sequence_len: true
 resize_token_embeddings_to_32x: false
 sample_packing: true
-save_steps: 40
-save_total_limit: 2
+save_steps: 200
+save_total_limit: 1
 sequence_len: 2048
-tokenizer_type: GPT2TokenizerFast
+tokenizer_type: LlamaTokenizerFast
 torch_dtype: bf16
 trust_remote_code: true
 val_set_size: 0.1
 wandb_entity: ''
 wandb_mode: online
-wandb_name: facebook/opt-125m-argilla/databricks-dolly-15k-curated-en
+wandb_name: TinyLlama/TinyLlama-1.1B-Chat-v1.0-argilla/databricks-dolly-15k-curated-en
 wandb_project: Gradients-On-Demand
 wandb_run: your_name
 wandb_runid: default
@@ -76,9 +76,7 @@ warmup_ratio: 0.05
 
 # test-repo
 
-This model is a fine-tuned version of [facebook/opt-125m](https://huggingface.co/facebook/opt-125m) on the argilla/databricks-dolly-15k-curated-en dataset.
-It achieves the following results on the evaluation set:
-- Loss: 3.0758
+This model is a fine-tuned version of [TinyLlama/TinyLlama-1.1B-Chat-v1.0](https://huggingface.co/TinyLlama/TinyLlama-1.1B-Chat-v1.0) on the argilla/databricks-dolly-15k-curated-en dataset.
 
 ## Model description
 
@@ -98,29 +96,23 @@ More information needed
 
 The following hyperparameters were used during training:
 - learning_rate: 0.0002
-- train_batch_size: 32
-- eval_batch_size: 32
+- train_batch_size: 12
+- eval_batch_size: 12
 - seed: 42
 - distributed_type: multi-GPU
 - num_devices: 8
-- total_train_batch_size: 256
-- total_eval_batch_size: 256
+- total_train_batch_size: 96
+- total_eval_batch_size: 96
 - optimizer: Use OptimizerNames.ADAMW_BNB with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
 - lr_scheduler_type: cosine
-- training_steps: 0
+- lr_scheduler_warmup_steps: 5
+- training_steps: 100
 
 ### Training results
 
-| Training Loss | Epoch   | Step | Validation Loss |
-|:-------------:|:-------:|:----:|:---------------:|
-| No log        | 0.3333  | 1    | 3.4280          |
-| 4.816         | 13.3333 | 40   | 3.2242          |
-| 3.9478        | 26.6667 | 80   | 3.1023          |
-| 3.4117        | 40.0    | 120  | 3.0508          |
-| 3.0986        | 53.3333 | 160  | 3.0802          |
-| 2.9113        | 66.6667 | 200  | 3.0881          |
-| 2.8333        | 80.0    | 240  | 3.0828          |
-| 2.8209        | 93.3333 | 280  | 3.0758          |
+| Training Loss | Epoch | Step | Validation Loss |
+|:-------------:|:-----:|:----:|:---------------:|
+| No log        | 0.125 | 1    | 2.0390          |
 
 
 ### Framework versions

generation_config.json

@@ -1,8 +1,8 @@
 {
-  "_from_model_config": true,
-  "bos_token_id": 2,
+  "bos_token_id": 1,
   "do_sample": true,
   "eos_token_id": 2,
-  "pad_token_id": 1,
+  "max_length": 2048,
+  "pad_token_id": 0,
   "transformers_version": "4.48.1"
 }