File size: 2,426 Bytes
290864c 13510ed 09674ea 290864c 4f8c26a 290864c 13510ed 290864c 13510ed 290864c 1de7f2c 290864c 4f8c26a 290864c 700adb6 290864c 4f8c26a 290864c 4f8c26a 290864c 16421ee 2a51e3d 290864c 1de7f2c 16421ee 290864c 1de7f2c fb6a3b1 290864c | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 | ---
library_name: peft
base_model: roberta-base
language:
- en
metrics:
- accuracy
pipeline_tag: text-classification
---
# Model Card for Model ID
<!-- Provide a quick summary of what the model is/does. -->
This model classifies whether the text/email is spam or ham.
## Model Details
This model is a fine-tuned version of roberta-base using LoRA specifically for a binary classification task containing
emails (subject + message) and a label whether it is spam or ham.
### Model Description
<!-- Provide a longer summary of what this model is. -->
The base model for this fine-tuning is roberta-base, which is a transformer-based model pre-trained on a large corpus of English data in a self-supervised fashion.
RoBERTa is an optimized version of BERT, designed to perform better on natural language understanding tasks.
We have applied LoRA to adapt the original RoBERTa model to the specific nuances of our binary classification problem.
LoRA introduces low-rank matrices that are trained during the fine-tuning process, enabling the model to learn task-specific
adaptations without altering the pre-trained weights directly.
- **Developed by:** Likhith231
- **Model type:** Text Classification
- **Finetuned from model:** Roberta Base
### Training Data
The model was finetuned on truncated version of SetFit/enron_spam dataset. The SetFit/enron_spam consists of 33716 rows.
The truncated version consists of 1000 train samples and 1000 test samples with columns, text and label.
### Model Sources
<!-- Provide the basic links for the model. -->
- **Repository:** https://huggingface.co/FacebookAI/roberta-base
## Training Details
- pretrained model= Roberta-base
- all params = 125,313,028
- trainable params= 665,858
- trainable% = 0.531355766137899
### Parameters
- weight_decay = 0.01
- lr = 1e-3
- batch_size = 4
- num_epochs = 10
### Results
Epoch|Training Loss|Validation Loss|Accuracy
-----|-------------|---------------|---------
1 |No log |0.172788 |0.957
2 |0.194500 |0.202991 |0.956
3 |0.194500 |0.229950 |0.958
4 |0.038400 |0.267390 |0.954
5 |0.038400 |0.283116 |0.963
6 |0.007000 |0.254960 |0.961
7 |0.007000 |0.299375 |0.961
8 |0.007900 |0.276321 |0.966
9 |0.007900 |0.275304 |0.967
10 |0.002000 |0.271234 |0.967
### Framework versions
- PEFT 0.8.2 |