| | --- |
| | tags: |
| | - generated_from_trainer |
| | datasets: |
| | - Hartunka/processed_book_corpus-km-5 |
| | metrics: |
| | - accuracy |
| | model-index: |
| | - name: tiny_bert_bc_km_5_v1 |
| | results: |
| | - task: |
| | name: Masked Language Modeling |
| | type: fill-mask |
| | dataset: |
| | name: Hartunka/processed_book_corpus-km-5 |
| | type: Hartunka/processed_book_corpus-km-5 |
| | metrics: |
| | - name: Accuracy |
| | type: accuracy |
| | value: 0.6796956039686967 |
| | --- |
| | |
| | <!-- This model card has been generated automatically according to the information the Trainer had access to. You |
| | should probably proofread and complete it, then remove this comment. --> |
| |
|
| | # tiny_bert_bc_km_5_v1 |
| | |
| | This model is a fine-tuned version of [](https://huggingface.co/) on the Hartunka/processed_book_corpus-km-5 dataset. |
| | It achieves the following results on the evaluation set: |
| | - Loss: 1.5993 |
| | - Accuracy: 0.6797 |
| | |
| | ## Model description |
| | |
| | More information needed |
| | |
| | ## Intended uses & limitations |
| | |
| | More information needed |
| | |
| | ## Training and evaluation data |
| | |
| | More information needed |
| | |
| | ## Training procedure |
| | |
| | ### Training hyperparameters |
| | |
| | The following hyperparameters were used during training: |
| | - learning_rate: 0.0001 |
| | - train_batch_size: 96 |
| | - eval_batch_size: 96 |
| | - seed: 10 |
| | - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08 |
| | - lr_scheduler_type: linear |
| | - lr_scheduler_warmup_steps: 10000 |
| | - num_epochs: 25 |
| |
|
| | ### Training results |
| |
|
| | | Training Loss | Epoch | Step | Validation Loss | Accuracy | |
| | |:-------------:|:-------:|:------:|:---------------:|:--------:| |
| | | 5.868 | 0.4215 | 10000 | 5.6390 | 0.1587 | |
| | | 5.757 | 0.8431 | 20000 | 5.5611 | 0.1631 | |
| | | 5.6406 | 1.2646 | 30000 | 5.4171 | 0.1782 | |
| | | 3.3457 | 1.6861 | 40000 | 2.9074 | 0.4865 | |
| | | 2.8544 | 2.1077 | 50000 | 2.4903 | 0.5423 | |
| | | 2.5821 | 2.5292 | 60000 | 2.2188 | 0.5786 | |
| | | 2.4361 | 2.9507 | 70000 | 2.0899 | 0.5988 | |
| | | 2.3431 | 3.3723 | 80000 | 2.0070 | 0.6118 | |
| | | 2.2752 | 3.7938 | 90000 | 1.9472 | 0.6216 | |
| | | 2.2181 | 4.2153 | 100000 | 1.9012 | 0.6284 | |
| | | 2.1874 | 4.6369 | 110000 | 1.8619 | 0.6334 | |
| | | 2.1438 | 5.0584 | 120000 | 1.8210 | 0.6391 | |
| | | 2.117 | 5.4799 | 130000 | 1.7991 | 0.6428 | |
| | | 2.1013 | 5.9014 | 140000 | 1.7722 | 0.6463 | |
| | | 2.0674 | 6.3230 | 150000 | 1.7556 | 0.6491 | |
| | | 2.0597 | 6.7445 | 160000 | 1.7411 | 0.6516 | |
| | | 2.0254 | 7.1660 | 170000 | 1.7298 | 0.6539 | |
| | | 2.0177 | 7.5876 | 180000 | 1.7182 | 0.6553 | |
| | | 2.0039 | 8.0091 | 190000 | 1.7304 | 0.6571 | |
| | | 1.9926 | 8.4306 | 200000 | 1.6948 | 0.6593 | |
| | | 1.9789 | 8.8522 | 210000 | 1.6922 | 0.6603 | |
| | | 1.9682 | 9.2737 | 220000 | 1.6792 | 0.6616 | |
| | | 1.9548 | 9.6952 | 230000 | 1.6715 | 0.6629 | |
| | | 1.9364 | 10.1168 | 240000 | 1.6794 | 0.6642 | |
| | | 1.9411 | 10.5383 | 250000 | 1.6578 | 0.6653 | |
| | | 1.9335 | 10.9598 | 260000 | 1.6482 | 0.6665 | |
| | | 1.9191 | 11.3814 | 270000 | 1.6511 | 0.6670 | |
| | | 1.9194 | 11.8029 | 280000 | 1.6416 | 0.6685 | |
| | | 1.9004 | 12.2244 | 290000 | 1.6558 | 0.6690 | |
| | | 1.8987 | 12.6460 | 300000 | 1.6351 | 0.6699 | |
| | | 1.8838 | 13.0675 | 310000 | 1.6363 | 0.6708 | |
| | | 1.8851 | 13.4890 | 320000 | 1.6329 | 0.6713 | |
| | | 1.8829 | 13.9106 | 330000 | 1.6368 | 0.6720 | |
| | | 1.8755 | 14.3321 | 340000 | 1.6218 | 0.6730 | |
| | | 1.8662 | 14.7536 | 350000 | 1.6259 | 0.6733 | |
| | | 1.8555 | 15.1751 | 360000 | 1.6301 | 0.6739 | |
| | | 1.8581 | 15.5967 | 370000 | 1.6150 | 0.6748 | |
| | | 1.8464 | 16.0182 | 380000 | 1.6229 | 0.6753 | |
| | | 1.8441 | 16.4397 | 390000 | 1.6136 | 0.6757 | |
| | | 1.8378 | 16.8613 | 400000 | 1.6066 | 0.6761 | |
| | | 1.8364 | 17.2828 | 410000 | 1.6130 | 0.6770 | |
| | | 1.8323 | 17.7043 | 420000 | 1.6023 | 0.6776 | |
| | | 1.8194 | 18.1259 | 430000 | 1.6100 | 0.6777 | |
| | | 1.8169 | 18.5474 | 440000 | 1.6132 | 0.6779 | |
| | | 1.8174 | 18.9689 | 450000 | 1.6051 | 0.6789 | |
| | | 1.8105 | 19.3905 | 460000 | 1.6220 | 0.6791 | |
| | | 1.8115 | 19.8120 | 470000 | 1.5991 | 0.6795 | |
| | | 1.8015 | 20.2335 | 480000 | 1.6119 | 0.6801 | |
| | | 1.7975 | 20.6551 | 490000 | 1.6027 | 0.6804 | |
| | | 1.7918 | 21.0766 | 500000 | 1.6094 | 0.6808 | |
| | | 1.7896 | 21.4981 | 510000 | 1.6055 | 0.6812 | |
| | | 1.7885 | 21.9197 | 520000 | 1.6038 | 0.6814 | |
| | | 1.7882 | 22.3412 | 530000 | 1.6068 | 0.6816 | |
| | | 1.7826 | 22.7627 | 540000 | 1.6105 | 0.6819 | |
| | | 1.7816 | 23.1843 | 550000 | 1.6100 | 0.6822 | |
| | | 1.7788 | 23.6058 | 560000 | 1.6041 | 0.6828 | |
| | | 1.772 | 24.0273 | 570000 | 1.6118 | 0.6827 | |
| | | 1.7736 | 24.4488 | 580000 | 1.6020 | 0.6835 | |
| | | 1.7698 | 24.8704 | 590000 | 1.6105 | 0.6830 | |
| |
|
| |
|
| | ### Framework versions |
| |
|
| | - Transformers 4.40.0 |
| | - Pytorch 2.6.0+cu124 |
| | - Datasets 3.5.0 |
| | - Tokenizers 0.19.1 |
| |
|
