End of training

README.md

@@ -111,21 +111,21 @@ save_steps: 200
 warmup_steps: 20
 eval_steps: 200
 
-sequence_len: 512
+sequence_len: 2048
 
-# false for exp
+# <<<< experimental settings <<<<
 sample_packing: false
-# true for exp
 train_on_inputs: true
+# >>>> experimental settings >>>
 
 pad_to_sequence_len: true
 
 gradient_accumulation_steps: 4
-micro_batch_size: 64
+micro_batch_size: 16
 
 optimizer: paged_adamw_8bit
 lr_scheduler: cosine
-learning_rate: 2e-5
+learning_rate: 1e-3
 
 bf16: auto
 tf32: false
@@ -157,7 +157,7 @@ weight_decay: 0.0
 
 This model is a fine-tuned version of [minpeter/pretrained-tiny-ko](https://huggingface.co/minpeter/pretrained-tiny-ko) on the lemon-mint/Korean-FineTome-100k, the lemon-mint/smol-koreantalk, the heegyu/open-korean-instructions-v20231020, the FreedomIntelligence/evol-instruct-korean, the FreedomIntelligence/alpaca-gpt4-korean, the FreedomIntelligence/sharegpt-korean, the coastral/korean-writing-style-instruct and the devngho/korean-instruction-mix datasets.
 It achieves the following results on the evaluation set:
-- Loss: 2.1993
+- Loss: 1.5699
 
 ## Model description
 
@@ -176,31 +176,44 @@ More information needed
 ### Training hyperparameters
 
 The following hyperparameters were used during training:
-- learning_rate: 2e-05
-- train_batch_size: 64
-- eval_batch_size: 64
+- learning_rate: 0.001
+- train_batch_size: 16
+- eval_batch_size: 16
 - seed: 42
 - distributed_type: multi-GPU
 - num_devices: 2
 - gradient_accumulation_steps: 4
-- total_train_batch_size: 512
-- total_eval_batch_size: 128
+- total_train_batch_size: 128
+- total_eval_batch_size: 32
 - optimizer: Use OptimizerNames.PAGED_ADAMW_8BIT with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
 - lr_scheduler_type: cosine
 - lr_scheduler_warmup_steps: 20
-- training_steps: 387
+- training_steps: 2972
 
 ### Training results
 
 | Training Loss | Epoch  | Step | Validation Loss |
 |:-------------:|:------:|:----:|:---------------:|
-| 4.2885        | 0.0078 | 1    | 4.3118          |
-| 2.1552        | 1.5504 | 200  | 2.1993          |
+| 2.8061        | 0.0010 | 1    | 2.8887          |
+| 1.9625        | 0.2019 | 200  | 1.9494          |
+| 1.8455        | 0.4037 | 400  | 1.8601          |
+| 1.7395        | 0.6056 | 600  | 1.8045          |
+| 1.7769        | 0.8075 | 800  | 1.7490          |
+| 1.5135        | 1.0091 | 1000 | 1.7116          |
+| 1.5928        | 1.2110 | 1200 | 1.6860          |
+| 1.5322        | 1.4128 | 1400 | 1.6517          |
+| 1.4939        | 1.6147 | 1600 | 1.6218          |
+| 1.4406        | 1.8166 | 1800 | 1.5939          |
+| 1.3999        | 2.0182 | 2000 | 1.5841          |
+| 1.3449        | 2.2200 | 2200 | 1.5770          |
+| 1.2352        | 2.4219 | 2400 | 1.5723          |
+| 1.3043        | 2.6238 | 2600 | 1.5702          |
+| 1.3467        | 2.8256 | 2800 | 1.5699          |
 
 
 ### Framework versions
 
-- Transformers 4.51.3
+- Transformers 4.52.3
 - Pytorch 2.6.0+cu124
-- Datasets 3.5.1
+- Datasets 3.6.0
 - Tokenizers 0.21.1