---
library_name: transformers
license: apache-2.0
base_model: distilbert-base-uncased
tags:
- generated_from_trainer
datasets:
- gokulsrinivasagan/processed_book_corpus-ld
metrics:
- accuracy
model-index:
- name: tinybert_train_book_v2
  results:
  - task:
      name: Masked Language Modeling
      type: fill-mask
    dataset:
      name: gokulsrinivasagan/processed_book_corpus-ld
      type: gokulsrinivasagan/processed_book_corpus-ld
    metrics:
    - name: Accuracy
      type: accuracy
      value: 0.6905521643636598
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# tinybert_train_book_v2

This model is a fine-tuned version of [distilbert-base-uncased](https://huggingface.co/distilbert-base-uncased) on the gokulsrinivasagan/processed_book_corpus-ld dataset.
It achieves the following results on the evaluation set:
- Loss: 1.4246
- Accuracy: 0.6906

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 0.0001
- train_batch_size: 160
- eval_batch_size: 160
- seed: 10
- optimizer: Use OptimizerNames.ADAMW_TORCH with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 10000
- num_epochs: 25

### Training results

| Training Loss | Epoch   | Step   | Validation Loss | Accuracy |
|:-------------:|:-------:|:------:|:---------------:|:--------:|
| 5.6396        | 0.7025  | 10000  | 5.4784          | 0.1645   |
| 5.5513        | 1.4051  | 20000  | 5.4083          | 0.1660   |
| 2.9289        | 2.1076  | 30000  | 2.5488          | 0.5214   |
| 2.5686        | 2.8102  | 40000  | 2.2269          | 0.5650   |
| 2.3542        | 3.5127  | 50000  | 2.0268          | 0.5945   |
| 2.2044        | 4.2153  | 60000  | 1.8924          | 0.6152   |
| 2.1102        | 4.9178  | 70000  | 1.8025          | 0.6296   |
| 2.0378        | 5.6203  | 80000  | 1.7395          | 0.6393   |
| 1.9881        | 6.3229  | 90000  | 1.6945          | 0.6467   |
| 1.9469        | 7.0254  | 100000 | 1.6552          | 0.6526   |
| 1.919         | 7.7280  | 110000 | 1.6279          | 0.6572   |
| 1.8899        | 8.4305  | 120000 | 1.6012          | 0.6613   |
| 1.8634        | 9.1331  | 130000 | 1.5849          | 0.6641   |
| 1.8481        | 9.8356  | 140000 | 1.5630          | 0.6679   |
| 1.8315        | 10.5381 | 150000 | 1.5476          | 0.6703   |
| 1.8209        | 11.2407 | 160000 | 1.5394          | 0.6716   |
| 1.8038        | 11.9432 | 170000 | 1.5250          | 0.6740   |
| 1.7932        | 12.6458 | 180000 | 1.5107          | 0.6761   |
| 1.7826        | 13.3483 | 190000 | 1.5058          | 0.6770   |
| 1.7736        | 14.0509 | 200000 | 1.4952          | 0.6785   |
| 1.7635        | 14.7534 | 210000 | 1.4862          | 0.6799   |
| 1.7549        | 15.4560 | 220000 | 1.4782          | 0.6815   |
| 1.7497        | 16.1585 | 230000 | 1.4737          | 0.6821   |
| 1.7407        | 16.8610 | 240000 | 1.4675          | 0.6832   |
| 1.7334        | 17.5636 | 250000 | 1.4612          | 0.6843   |
| 1.7288        | 18.2661 | 260000 | 1.4568          | 0.6849   |
| 1.7265        | 18.9687 | 270000 | 1.4519          | 0.6858   |
| 1.7179        | 19.6712 | 280000 | 1.4469          | 0.6868   |
| 1.7156        | 20.3738 | 290000 | 1.4428          | 0.6873   |
| 1.7086        | 21.0763 | 300000 | 1.4389          | 0.6882   |
| 1.7064        | 21.7788 | 310000 | 1.4334          | 0.6892   |
| 1.7009        | 22.4814 | 320000 | 1.4309          | 0.6895   |
| 1.6989        | 23.1839 | 330000 | 1.4292          | 0.6896   |
| 1.6971        | 23.8865 | 340000 | 1.4262          | 0.6902   |
| 1.6958        | 24.5890 | 350000 | 1.4239          | 0.6906   |


### Framework versions

- Transformers 4.46.1
- Pytorch 2.2.0+cu121
- Datasets 3.1.0
- Tokenizers 0.20.1