24679-HW2-text-distilbert-predictor

This model is a fine-tuned version of distilbert-base-uncased on the None dataset. It achieves the following results on the evaluation set:

• Loss: 0.0008
• Accuracy: 1.0
• F1: 1.0
• Precision: 1.0
• Recall: 1.0

Model description

This model uses the dataset jennifee/HW1-aug-text-dataset which contains a text Review of a book, and classifies it as Fiction (1) or Nonfiction(0). The model seeks to fine-tune DistilBERT to accomplish this classification task. It computes a confusion matrix and a brief error analysis of several misclassifications.

Intended uses & limitations

This is intended to perform and demonstrate finetuning a text model. It is trained on a relatively small dataset and so may not be applicable for large scale applications.

Training and evaluation data

This dataset contains a text 'Review' of a book, and classifies it as Fiction (1) or Nonfiction(0) It contains two splits of original (100 rows) and augmented (1.6k rows) The augmented dataset is split into train/validation/test sets using an 80/20 split

Training procedure

The model is tokenized for binary classifiation such that train: 1024 | val: 256 | test: 320 | ext_valid: 100

The hypermaraters for training are listed below

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 2e-05
• train_batch_size: 8
• eval_batch_size: 8
• seed: 42
• optimizer: Use OptimizerNames.ADAMW_TORCH_FUSED with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: linear
• num_epochs: 5

Training results

Training Loss	Epoch	Step	Validation Loss	Accuracy	F1	Precision	Recall
0.005	1.0	128	0.0343	0.9922	0.9922	0.9923	0.9922
0.002	2.0	256	0.0072	0.9961	0.9961	0.9961	0.9961
0.0011	3.0	384	0.0013	1.0	1.0	1.0	1.0
0.0008	4.0	512	0.0007	1.0	1.0	1.0	1.0
0.0016	5.0	640	0.0006	1.0	1.0	1.0	1.0

Framework versions

• Transformers 4.56.1
• Pytorch 2.8.0+cu126
• Datasets 4.0.0
• Tokenizers 0.22.0