--- library_name: transformers tags: - trl - sft - generated_from_trainer model-index: - name: llama_3b_step2_batch_v1 results: [] --- # llama_3b_step2_batch_v1 This model was trained from scratch on an unknown dataset. It achieves the following results on the evaluation set: - Loss: 0.5060 ## Model description More information needed ## Intended uses & limitations More information needed ## Training and evaluation data More information needed ## Training procedure ### Training hyperparameters The following hyperparameters were used during training: - learning_rate: 1e-05 - train_batch_size: 1 - eval_batch_size: 40 - seed: 42 - gradient_accumulation_steps: 4 - total_train_batch_size: 4 - optimizer: Use adamw_torch with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments - lr_scheduler_type: linear - num_epochs: 2 ### Training results | Training Loss | Epoch | Step | Validation Loss | |:-------------:|:------:|:----:|:---------------:| | 1.0531 | 0.0170 | 50 | 1.2007 | | 1.0336 | 0.0341 | 100 | 1.1242 | | 0.9428 | 0.0511 | 150 | 1.0800 | | 1.4386 | 0.0682 | 200 | 1.0408 | | 0.8375 | 0.0852 | 250 | 1.0127 | | 0.9193 | 0.1023 | 300 | 0.9817 | | 1.0368 | 0.1193 | 350 | 0.9573 | | 1.2018 | 0.1364 | 400 | 0.9319 | | 1.2749 | 0.1534 | 450 | 0.9072 | | 0.9881 | 0.1704 | 500 | 0.8820 | | 0.9707 | 0.1875 | 550 | 0.8599 | | 1.2377 | 0.2045 | 600 | 0.8412 | | 0.9024 | 0.2216 | 650 | 0.8180 | | 0.5889 | 0.2386 | 700 | 0.8024 | | 0.8046 | 0.2557 | 750 | 0.7899 | | 0.83 | 0.2727 | 800 | 0.7710 | | 0.6852 | 0.2898 | 850 | 0.7548 | | 0.8512 | 0.3068 | 900 | 0.7422 | | 0.8377 | 0.3238 | 950 | 0.7345 | | 0.5361 | 0.3409 | 1000 | 0.7220 | | 0.7696 | 0.3579 | 1050 | 0.7105 | | 0.8175 | 0.3750 | 1100 | 0.7013 | | 0.6144 | 0.3920 | 1150 | 0.6886 | | 0.3598 | 0.4091 | 1200 | 0.6809 | | 0.7176 | 0.4261 | 1250 | 0.6692 | | 0.5281 | 0.4432 | 1300 | 0.6644 | | 0.3555 | 0.4602 | 1350 | 0.6547 | | 0.9024 | 0.4772 | 1400 | 0.6471 | | 0.7713 | 0.4943 | 1450 | 0.6386 | | 0.6172 | 0.5113 | 1500 | 0.6322 | | 0.6325 | 0.5284 | 1550 | 0.6266 | | 0.7503 | 0.5454 | 1600 | 0.6206 | | 0.349 | 0.5625 | 1650 | 0.6136 | | 0.7 | 0.5795 | 1700 | 0.6085 | | 0.5014 | 0.5966 | 1750 | 0.6023 | | 0.6441 | 0.6136 | 1800 | 0.5975 | | 0.5066 | 0.6306 | 1850 | 0.5921 | | 0.6036 | 0.6477 | 1900 | 0.5883 | | 0.6549 | 0.6647 | 1950 | 0.5840 | | 0.3903 | 0.6818 | 2000 | 0.5789 | | 0.8864 | 0.6988 | 2050 | 0.5754 | | 0.7164 | 0.7159 | 2100 | 0.5709 | | 0.5504 | 0.7329 | 2150 | 0.5687 | | 0.4216 | 0.7500 | 2200 | 0.5646 | | 0.4241 | 0.7670 | 2250 | 0.5618 | | 0.6452 | 0.7840 | 2300 | 0.5590 | | 0.7067 | 0.8011 | 2350 | 0.5558 | | 0.4536 | 0.8181 | 2400 | 0.5537 | | 0.8657 | 0.8352 | 2450 | 0.5508 | | 0.7452 | 0.8522 | 2500 | 0.5483 | | 0.3444 | 0.8693 | 2550 | 0.5458 | | 0.2889 | 0.8863 | 2600 | 0.5437 | | 0.2415 | 0.9034 | 2650 | 0.5401 | | 0.5393 | 0.9204 | 2700 | 0.5385 | | 0.4866 | 0.9374 | 2750 | 0.5372 | | 0.9233 | 0.9545 | 2800 | 0.5347 | | 0.4623 | 0.9715 | 2850 | 0.5318 | | 0.4211 | 0.9886 | 2900 | 0.5299 | | 0.4308 | 1.0056 | 2950 | 0.5283 | | 0.618 | 1.0227 | 3000 | 0.5285 | | 0.7693 | 1.0397 | 3050 | 0.5262 | | 0.2893 | 1.0568 | 3100 | 0.5266 | | 0.461 | 1.0738 | 3150 | 0.5273 | | 0.3648 | 1.0908 | 3200 | 0.5230 | | 0.4981 | 1.1079 | 3250 | 0.5253 | | 0.5005 | 1.1249 | 3300 | 0.5222 | | 0.4117 | 1.1420 | 3350 | 0.5217 | | 0.3319 | 1.1590 | 3400 | 0.5188 | | 0.2549 | 1.1761 | 3450 | 0.5190 | | 0.3758 | 1.1931 | 3500 | 0.5186 | | 0.2889 | 1.2102 | 3550 | 0.5173 | | 0.6341 | 1.2272 | 3600 | 0.5167 | | 0.3217 | 1.2442 | 3650 | 0.5155 | | 0.4406 | 1.2613 | 3700 | 0.5150 | | 0.7445 | 1.2783 | 3750 | 0.5148 | | 0.5511 | 1.2954 | 3800 | 0.5133 | | 0.3933 | 1.3124 | 3850 | 0.5125 | | 0.39 | 1.3295 | 3900 | 0.5134 | | 0.3015 | 1.3465 | 3950 | 0.5126 | | 0.8124 | 1.3636 | 4000 | 0.5118 | | 0.6512 | 1.3806 | 4050 | 0.5111 | | 0.7011 | 1.3976 | 4100 | 0.5106 | | 0.4556 | 1.4147 | 4150 | 0.5103 | | 0.4563 | 1.4317 | 4200 | 0.5100 | | 0.2651 | 1.4488 | 4250 | 0.5100 | | 0.5674 | 1.4658 | 4300 | 0.5090 | | 0.2869 | 1.4829 | 4350 | 0.5093 | | 0.5327 | 1.4999 | 4400 | 0.5088 | | 0.726 | 1.5170 | 4450 | 0.5086 | | 0.2619 | 1.5340 | 4500 | 0.5084 | | 0.6597 | 1.5510 | 4550 | 0.5081 | | 0.4848 | 1.5681 | 4600 | 0.5083 | | 0.412 | 1.5851 | 4650 | 0.5080 | | 0.6712 | 1.6022 | 4700 | 0.5077 | | 0.5523 | 1.6192 | 4750 | 0.5076 | | 0.5105 | 1.6363 | 4800 | 0.5077 | | 0.5315 | 1.6533 | 4850 | 0.5071 | | 0.4166 | 1.6704 | 4900 | 0.5069 | | 0.4081 | 1.6874 | 4950 | 0.5065 | | 0.3154 | 1.7044 | 5000 | 0.5063 | | 0.396 | 1.7215 | 5050 | 0.5063 | | 0.6121 | 1.7385 | 5100 | 0.5064 | | 0.379 | 1.7556 | 5150 | 0.5063 | | 0.4534 | 1.7726 | 5200 | 0.5061 | | 0.5572 | 1.7897 | 5250 | 0.5060 | | 0.3847 | 1.8067 | 5300 | 0.5059 | | 0.3751 | 1.8238 | 5350 | 0.5060 | | 0.4346 | 1.8408 | 5400 | 0.5061 | | 0.4928 | 1.8578 | 5450 | 0.5061 | | 0.5215 | 1.8749 | 5500 | 0.5060 | | 0.6156 | 1.8919 | 5550 | 0.5060 | | 0.4041 | 1.9090 | 5600 | 0.5060 | | 0.5604 | 1.9260 | 5650 | 0.5059 | | 0.424 | 1.9431 | 5700 | 0.5060 | | 0.1856 | 1.9601 | 5750 | 0.5060 | | 0.3701 | 1.9772 | 5800 | 0.5061 | | 0.4201 | 1.9942 | 5850 | 0.5060 | ### Framework versions - Transformers 4.46.1 - Pytorch 2.1.0+cu118 - Datasets 3.0.2 - Tokenizers 0.20.1