End of training

README.md

@@ -50,12 +50,12 @@ save_steps: 200
 warmup_steps: 100
 eval_steps: 200
 
-sequence_len: 512
+sequence_len: 1024
 sample_packing: true
 pad_to_sequence_len: true
 
 gradient_accumulation_steps: 4
-micro_batch_size: 56
+micro_batch_size: 32
 
 optimizer: paged_adamw_8bit
 lr_scheduler: cosine
@@ -91,7 +91,7 @@ weight_decay: 0.0
 
 This model is a fine-tuned version of [minpeter/pretrained-tiny-ko](https://huggingface.co/minpeter/pretrained-tiny-ko) on the lemon-mint/Korean-FineTome-100k and the lemon-mint/smol-koreantalk datasets.
 It achieves the following results on the evaluation set:
-- Loss: 3.6623
+- Loss: 3.6038
 
 ## Model description
 
@@ -111,24 +111,24 @@ More information needed
 
 The following hyperparameters were used during training:
 - learning_rate: 2e-05
-- train_batch_size: 56
-- eval_batch_size: 56
+- train_batch_size: 32
+- eval_batch_size: 32
 - seed: 42
 - distributed_type: multi-GPU
 - num_devices: 4
 - gradient_accumulation_steps: 4
-- total_train_batch_size: 896
-- total_eval_batch_size: 224
+- total_train_batch_size: 512
+- total_eval_batch_size: 128
 - optimizer: Use OptimizerNames.PAGED_ADAMW_8BIT with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
 - lr_scheduler_type: cosine
 - lr_scheduler_warmup_steps: 100
-- training_steps: 48
+- training_steps: 102
 
 ### Training results
 
 | Training Loss | Epoch  | Step | Validation Loss |
 |:-------------:|:------:|:----:|:---------------:|
-| 3.6868        | 0.0404 | 1    | 3.6623          |
+| 3.5674        | 0.0193 | 1    | 3.6038          |
 
 
 ### Framework versions