MiniLMv2-L6-H384_R-OCR-quality

This model is a fine-tuned version of pszemraj/MiniLMv2-L6-H384_R-fineweb-100k on pszemraj/OCR-quality-classification It achieves the following results on the evaluation set:

• Loss: 0.0162
• Accuracy: 0.996
• Num Input Tokens Seen: 61536256

Intended uses & limitations

predict whether a document is clean or noisy

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 3e-05
• train_batch_size: 8
• eval_batch_size: 8
• seed: 42
• gradient_accumulation_steps: 8
• total_train_batch_size: 64
• optimizer: Adam with betas=(0.9,0.99) and epsilon=1e-08
• lr_scheduler_type: linear
• lr_scheduler_warmup_ratio: 0.05
• num_epochs: 2.0

Training results

Training Loss	Epoch	Step	Validation Loss	Accuracy	Input Tokens Seen
0.0298	0.2660	250	0.0448	0.99	8192000
0.0141	0.5321	500	0.0330	0.99	16384000
0.02	0.7981	750	0.0298	0.99	24576000
0.0085	1.0641	1000	0.0222	0.994	32765952
0.0174	1.3301	1250	0.0207	0.994	40957952
0.0104	1.5962	1500	0.0202	0.996	49149952
0.0237	1.8622	1750	0.0185	0.996	57341952

Framework versions

• Transformers 4.40.2
• Pytorch 2.2.0+cu121
• Datasets 2.19.1
• Tokenizers 0.19.1