End of training

README.md

@@ -1,7 +1,6 @@
 ---
 library_name: transformers
-license: apache-2.0
-base_model: TinyLlama/TinyLlama-1.1B-Chat-v1.0
+base_model: trl-internal-testing/tiny-random-LlamaForCausalLM
 tags:
 - axolotl
 - generated_from_trainer
@@ -20,8 +19,8 @@ should probably proofread and complete it, then remove this comment. -->
 
 axolotl version: `0.6.0`
 ```yaml
-base_model: TinyLlama/TinyLlama-1.1B-Chat-v1.0
-batch_size: 96
+base_model: trl-internal-testing/tiny-random-LlamaForCausalLM
+batch_size: 256
 bf16: true
 chat_template: tokenizer_default_fallback_alpaca
 datasets:
@@ -47,7 +46,7 @@ learning_rate: 0.0002
 logging_steps: 10
 lr_scheduler: cosine
 max_steps: 20000
-micro_batch_size: 12
+micro_batch_size: 32
 model_type: AutoModelForCausalLM
 num_epochs: 100
 optimizer: adamw_bnb_8bit
@@ -64,7 +63,7 @@ trust_remote_code: true
 val_set_size: 0.1
 wandb_entity: ''
 wandb_mode: online
-wandb_name: TinyLlama/TinyLlama-1.1B-Chat-v1.0-argilla/databricks-dolly-15k-curated-en
+wandb_name: trl-internal-testing/tiny-random-LlamaForCausalLM-argilla/databricks-dolly-15k-curated-en
 wandb_project: Gradients-On-Demand
 wandb_run: your_name
 wandb_runid: default
@@ -76,7 +75,9 @@ warmup_ratio: 0.05
 
 # test-repo
 
-This model is a fine-tuned version of [TinyLlama/TinyLlama-1.1B-Chat-v1.0](https://huggingface.co/TinyLlama/TinyLlama-1.1B-Chat-v1.0) on the argilla/databricks-dolly-15k-curated-en dataset.
+This model is a fine-tuned version of [trl-internal-testing/tiny-random-LlamaForCausalLM](https://huggingface.co/trl-internal-testing/tiny-random-LlamaForCausalLM) on the argilla/databricks-dolly-15k-curated-en dataset.
+It achieves the following results on the evaluation set:
+- Loss: 9.6817
 
 ## Model description
 
@@ -96,23 +97,23 @@ More information needed
 
 The following hyperparameters were used during training:
 - learning_rate: 0.0002
-- train_batch_size: 12
-- eval_batch_size: 12
+- train_batch_size: 32
+- eval_batch_size: 32
 - seed: 42
 - distributed_type: multi-GPU
 - num_devices: 8
-- total_train_batch_size: 96
-- total_eval_batch_size: 96
+- total_train_batch_size: 256
+- total_eval_batch_size: 256
 - optimizer: Use OptimizerNames.ADAMW_BNB with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
 - lr_scheduler_type: cosine
-- lr_scheduler_warmup_steps: 5
-- training_steps: 100
+- training_steps: 0
 
 ### Training results
 
-| Training Loss | Epoch | Step | Validation Loss |
-|:-------------:|:-----:|:----:|:---------------:|
-| No log        | 0.125 | 1    | 2.0390          |
+| Training Loss | Epoch   | Step | Validation Loss |
+|:-------------:|:-------:|:----:|:---------------:|
+| No log        | 0.3333  | 1    | 10.3756         |
+| 9.6746        | 66.6667 | 200  | 9.6817          |
 
 
 ### Framework versions

generation_config.json

@@ -1,8 +1,8 @@
 {
-  "bos_token_id": 1,
+  "_from_model_config": true,
+  "bos_token_id": 0,
   "do_sample": true,
-  "eos_token_id": 2,
-  "max_length": 2048,
-  "pad_token_id": 0,
+  "eos_token_id": 1,
+  "pad_token_id": 2,
   "transformers_version": "4.48.1"
 }