---
tags:
- generated_from_trainer
datasets:
- Hartunka/processed_book_corpus-km-5
metrics:
- accuracy
model-index:
- name: tiny_bert_bc_km_5_v1
  results:
  - task:
      name: Masked Language Modeling
      type: fill-mask
    dataset:
      name: Hartunka/processed_book_corpus-km-5
      type: Hartunka/processed_book_corpus-km-5
    metrics:
    - name: Accuracy
      type: accuracy
      value: 0.6796956039686967
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# tiny_bert_bc_km_5_v1

This model is a fine-tuned version of [](https://huggingface.co/) on the Hartunka/processed_book_corpus-km-5 dataset.
It achieves the following results on the evaluation set:
- Loss: 1.5993
- Accuracy: 0.6797

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 0.0001
- train_batch_size: 96
- eval_batch_size: 96
- seed: 10
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 10000
- num_epochs: 25

### Training results

| Training Loss | Epoch   | Step   | Validation Loss | Accuracy |
|:-------------:|:-------:|:------:|:---------------:|:--------:|
| 5.868         | 0.4215  | 10000  | 5.6390          | 0.1587   |
| 5.757         | 0.8431  | 20000  | 5.5611          | 0.1631   |
| 5.6406        | 1.2646  | 30000  | 5.4171          | 0.1782   |
| 3.3457        | 1.6861  | 40000  | 2.9074          | 0.4865   |
| 2.8544        | 2.1077  | 50000  | 2.4903          | 0.5423   |
| 2.5821        | 2.5292  | 60000  | 2.2188          | 0.5786   |
| 2.4361        | 2.9507  | 70000  | 2.0899          | 0.5988   |
| 2.3431        | 3.3723  | 80000  | 2.0070          | 0.6118   |
| 2.2752        | 3.7938  | 90000  | 1.9472          | 0.6216   |
| 2.2181        | 4.2153  | 100000 | 1.9012          | 0.6284   |
| 2.1874        | 4.6369  | 110000 | 1.8619          | 0.6334   |
| 2.1438        | 5.0584  | 120000 | 1.8210          | 0.6391   |
| 2.117         | 5.4799  | 130000 | 1.7991          | 0.6428   |
| 2.1013        | 5.9014  | 140000 | 1.7722          | 0.6463   |
| 2.0674        | 6.3230  | 150000 | 1.7556          | 0.6491   |
| 2.0597        | 6.7445  | 160000 | 1.7411          | 0.6516   |
| 2.0254        | 7.1660  | 170000 | 1.7298          | 0.6539   |
| 2.0177        | 7.5876  | 180000 | 1.7182          | 0.6553   |
| 2.0039        | 8.0091  | 190000 | 1.7304          | 0.6571   |
| 1.9926        | 8.4306  | 200000 | 1.6948          | 0.6593   |
| 1.9789        | 8.8522  | 210000 | 1.6922          | 0.6603   |
| 1.9682        | 9.2737  | 220000 | 1.6792          | 0.6616   |
| 1.9548        | 9.6952  | 230000 | 1.6715          | 0.6629   |
| 1.9364        | 10.1168 | 240000 | 1.6794          | 0.6642   |
| 1.9411        | 10.5383 | 250000 | 1.6578          | 0.6653   |
| 1.9335        | 10.9598 | 260000 | 1.6482          | 0.6665   |
| 1.9191        | 11.3814 | 270000 | 1.6511          | 0.6670   |
| 1.9194        | 11.8029 | 280000 | 1.6416          | 0.6685   |
| 1.9004        | 12.2244 | 290000 | 1.6558          | 0.6690   |
| 1.8987        | 12.6460 | 300000 | 1.6351          | 0.6699   |
| 1.8838        | 13.0675 | 310000 | 1.6363          | 0.6708   |
| 1.8851        | 13.4890 | 320000 | 1.6329          | 0.6713   |
| 1.8829        | 13.9106 | 330000 | 1.6368          | 0.6720   |
| 1.8755        | 14.3321 | 340000 | 1.6218          | 0.6730   |
| 1.8662        | 14.7536 | 350000 | 1.6259          | 0.6733   |
| 1.8555        | 15.1751 | 360000 | 1.6301          | 0.6739   |
| 1.8581        | 15.5967 | 370000 | 1.6150          | 0.6748   |
| 1.8464        | 16.0182 | 380000 | 1.6229          | 0.6753   |
| 1.8441        | 16.4397 | 390000 | 1.6136          | 0.6757   |
| 1.8378        | 16.8613 | 400000 | 1.6066          | 0.6761   |
| 1.8364        | 17.2828 | 410000 | 1.6130          | 0.6770   |
| 1.8323        | 17.7043 | 420000 | 1.6023          | 0.6776   |
| 1.8194        | 18.1259 | 430000 | 1.6100          | 0.6777   |
| 1.8169        | 18.5474 | 440000 | 1.6132          | 0.6779   |
| 1.8174        | 18.9689 | 450000 | 1.6051          | 0.6789   |
| 1.8105        | 19.3905 | 460000 | 1.6220          | 0.6791   |
| 1.8115        | 19.8120 | 470000 | 1.5991          | 0.6795   |
| 1.8015        | 20.2335 | 480000 | 1.6119          | 0.6801   |
| 1.7975        | 20.6551 | 490000 | 1.6027          | 0.6804   |
| 1.7918        | 21.0766 | 500000 | 1.6094          | 0.6808   |
| 1.7896        | 21.4981 | 510000 | 1.6055          | 0.6812   |
| 1.7885        | 21.9197 | 520000 | 1.6038          | 0.6814   |
| 1.7882        | 22.3412 | 530000 | 1.6068          | 0.6816   |
| 1.7826        | 22.7627 | 540000 | 1.6105          | 0.6819   |
| 1.7816        | 23.1843 | 550000 | 1.6100          | 0.6822   |
| 1.7788        | 23.6058 | 560000 | 1.6041          | 0.6828   |
| 1.772         | 24.0273 | 570000 | 1.6118          | 0.6827   |
| 1.7736        | 24.4488 | 580000 | 1.6020          | 0.6835   |
| 1.7698        | 24.8704 | 590000 | 1.6105          | 0.6830   |


### Framework versions

- Transformers 4.40.0
- Pytorch 2.6.0+cu124
- Datasets 3.5.0
- Tokenizers 0.19.1