---
library_name: transformers
base_model: huawei-noah/TinyBERT_General_4L_312D
tags:
- generated_from_trainer
metrics:
- accuracy
- precision
- recall
model-index:
- name: Structured-NF4-KD-NID
  results: []
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# Structured-NF4-KD-NID

This model is a fine-tuned version of [huawei-noah/TinyBERT_General_4L_312D](https://huggingface.co/huawei-noah/TinyBERT_General_4L_312D) on an unknown dataset.
It achieves the following results on the evaluation set:
- Loss: 0.0384
- Accuracy: 0.9923
- Precision: 0.9796
- Recall: 0.9374
- F1 score: 0.9470

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 5e-05
- train_batch_size: 650
- eval_batch_size: 650
- seed: 42
- optimizer: Use OptimizerNames.ADAMW_TORCH with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
- lr_scheduler_type: linear
- num_epochs: 20
- mixed_precision_training: Native AMP

### Training results

| Training Loss | Epoch | Step  | Validation Loss | Accuracy | Precision | Recall | F1 score |
|:-------------:|:-----:|:-----:|:---------------:|:--------:|:---------:|:------:|:--------:|
| 0.217         | 1.0   | 1828  | 0.1859          | 0.9775   | 0.7808    | 0.7811 | 0.7797   |
| 0.1295        | 2.0   | 3656  | 0.1142          | 0.9844   | 0.9142    | 0.8421 | 0.8404   |
| 0.0958        | 3.0   | 5484  | 0.0916          | 0.9864   | 0.9136    | 0.8662 | 0.8699   |
| 0.0846        | 4.0   | 7312  | 0.0779          | 0.9882   | 0.9207    | 0.8902 | 0.8953   |
| 0.0683        | 5.0   | 9140  | 0.0677          | 0.9894   | 0.9246    | 0.9034 | 0.9073   |
| 0.07          | 6.0   | 10968 | 0.0599          | 0.9899   | 0.9259    | 0.9103 | 0.9118   |
| 0.0574        | 7.0   | 12796 | 0.0555          | 0.9904   | 0.9284    | 0.9097 | 0.9156   |
| 0.0503        | 8.0   | 14624 | 0.0541          | 0.9906   | 0.9211    | 0.9185 | 0.9162   |
| 0.048         | 9.0   | 16452 | 0.0498          | 0.9911   | 0.9318    | 0.9185 | 0.9217   |
| 0.0479        | 10.0  | 18280 | 0.0483          | 0.9912   | 0.9353    | 0.9178 | 0.9232   |
| 0.0452        | 11.0  | 20108 | 0.0467          | 0.9914   | 0.9264    | 0.9233 | 0.9213   |
| 0.042         | 12.0  | 21936 | 0.0437          | 0.9917   | 0.9331    | 0.9211 | 0.9244   |
| 0.038         | 13.0  | 23764 | 0.0426          | 0.9918   | 0.9782    | 0.9282 | 0.9366   |
| 0.0358        | 14.0  | 25592 | 0.0414          | 0.9919   | 0.9359    | 0.9226 | 0.9253   |
| 0.0351        | 15.0  | 27420 | 0.0417          | 0.9920   | 0.9785    | 0.9288 | 0.9370   |
| 0.0319        | 16.0  | 29248 | 0.0404          | 0.9921   | 0.9768    | 0.9437 | 0.9518   |
| 0.0302        | 17.0  | 31076 | 0.0391          | 0.9922   | 0.9789    | 0.9319 | 0.9386   |
| 0.0319        | 18.0  | 32904 | 0.0383          | 0.9922   | 0.9801    | 0.9428 | 0.9533   |
| 0.0265        | 19.0  | 34732 | 0.0383          | 0.9923   | 0.9802    | 0.9436 | 0.9542   |
| 0.0277        | 20.0  | 36560 | 0.0384          | 0.9923   | 0.9796    | 0.9374 | 0.9470   |


### Framework versions

- Transformers 4.50.0
- Pytorch 2.6.0+cu124
- Datasets 3.4.1
- Tokenizers 0.21.1