End of training

README.md

@@ -20,7 +20,7 @@ should probably proofread and complete it, then remove this comment. -->
 axolotl version: `0.6.0`
 ```yaml
 base_model: trl-internal-testing/tiny-random-LlamaForCausalLM
-batch_size: 64
+batch_size: 128
 bf16: true
 chat_template: tokenizer_default_fallback_alpaca
 datasets:
@@ -36,7 +36,7 @@ datasets:
     system_prompt: ''
 device_map: auto
 eval_sample_packing: false
-eval_steps: 50
+eval_steps: 200
 flash_attention: true
 gradient_checkpointing: true
 group_by_length: true
@@ -45,7 +45,7 @@ hub_strategy: checkpoint
 learning_rate: 0.0002
 logging_steps: 10
 lr_scheduler: cosine
-max_steps: 5000
+max_steps: 10000
 micro_batch_size: 32
 model_type: AutoModelForCausalLM
 num_epochs: 100
@@ -54,11 +54,13 @@ output_dir: /root/.sn56/axolotl/tmp/tiny-random-LlamaForCausalLM
 pad_to_sequence_len: true
 resize_token_embeddings_to_32x: false
 sample_packing: true
-save_steps: 50
-save_total_limit: 2
+save_steps: 200
+save_total_limit: 1
 sequence_len: 2048
 tokenizer_type: LlamaTokenizerFast
 torch_dtype: bf16
+training_args_kwargs:
+  hub_private_repo: true
 trust_remote_code: true
 val_set_size: 0.1
 wandb_entity: ''
@@ -76,8 +78,6 @@ warmup_ratio: 0.05
 # tiny-random-LlamaForCausalLM
 
 This model is a fine-tuned version of [trl-internal-testing/tiny-random-LlamaForCausalLM](https://huggingface.co/trl-internal-testing/tiny-random-LlamaForCausalLM) on the argilla/databricks-dolly-15k-curated-en dataset.
-It achieves the following results on the evaluation set:
-- Loss: 9.1943
 
 ## Model description
 
@@ -101,31 +101,19 @@ The following hyperparameters were used during training:
 - eval_batch_size: 32
 - seed: 42
 - distributed_type: multi-GPU
-- num_devices: 2
-- total_train_batch_size: 64
-- total_eval_batch_size: 64
+- num_devices: 4
+- total_train_batch_size: 128
+- total_eval_batch_size: 128
 - optimizer: Use OptimizerNames.ADAMW_BNB with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
 - lr_scheduler_type: cosine
-- lr_scheduler_warmup_steps: 30
-- training_steps: 600
+- lr_scheduler_warmup_steps: 5
+- training_steps: 100
 
 ### Training results
 
-| Training Loss | Epoch   | Step | Validation Loss |
-|:-------------:|:-------:|:----:|:---------------:|
-| No log        | 0.0769  | 1    | 10.3764         |
-| 10.3159       | 3.8462  | 50   | 10.2852         |
-| 9.998         | 7.6923  | 100  | 9.9738          |
-| 9.7359        | 11.5385 | 150  | 9.7190          |
-| 9.5151        | 15.3846 | 200  | 9.5042          |
-| 9.3407        | 19.2308 | 250  | 9.3411          |
-| 9.2338        | 23.0769 | 300  | 9.2415          |
-| 9.1896        | 26.9231 | 350  | 9.2039          |
-| 9.18          | 30.7692 | 400  | 9.1960          |
-| 9.1777        | 34.6154 | 450  | 9.1957          |
-| 9.1781        | 38.4615 | 500  | 9.1931          |
-| 9.1761        | 42.3077 | 550  | 9.1936          |
-| 9.1762        | 46.1538 | 600  | 9.1943          |
+| Training Loss | Epoch  | Step | Validation Loss |
+|:-------------:|:------:|:----:|:---------------:|
+| No log        | 0.1667 | 1    | 10.3764         |
 
 
 ### Framework versions