| --- |
| library_name: peft |
| base_model: roberta-base |
| language: |
| - en |
| metrics: |
| - accuracy |
| pipeline_tag: text-classification |
| --- |
| |
| # Model Card for Model ID |
|
|
| <!-- Provide a quick summary of what the model is/does. --> |
| This model classifies whether the text/email is spam or ham. |
|
|
| ## Model Details |
| This model is a fine-tuned version of roberta-base using LoRA specifically for a binary classification task containing |
| emails (subject + message) and a label whether it is spam or ham. |
|
|
| ### Model Description |
|
|
| <!-- Provide a longer summary of what this model is. --> |
| The base model for this fine-tuning is roberta-base, which is a transformer-based model pre-trained on a large corpus of English data in a self-supervised fashion. |
| RoBERTa is an optimized version of BERT, designed to perform better on natural language understanding tasks. |
| We have applied LoRA to adapt the original RoBERTa model to the specific nuances of our binary classification problem. |
| LoRA introduces low-rank matrices that are trained during the fine-tuning process, enabling the model to learn task-specific |
| adaptations without altering the pre-trained weights directly. |
|
|
| - **Developed by:** Likhith231 |
| - **Model type:** Text Classification |
| - **Finetuned from model:** Roberta Base |
|
|
| ### Training Data |
| The model was finetuned on truncated version of SetFit/enron_spam dataset. The SetFit/enron_spam consists of 33716 rows. |
| The truncated version consists of 1000 train samples and 1000 test samples with columns, text and label. |
|
|
|
|
| ### Model Sources |
|
|
| <!-- Provide the basic links for the model. --> |
|
|
| - **Repository:** https://huggingface.co/FacebookAI/roberta-base |
|
|
| ## Training Details |
|
|
| - pretrained model= Roberta-base |
| - all params = 125,313,028 |
| - trainable params= 665,858 |
| - trainable% = 0.531355766137899 |
|
|
| ### Parameters |
| - weight_decay = 0.01 |
| - lr = 1e-3 |
| - batch_size = 4 |
| - num_epochs = 10 |
| |
| ### Results |
| |
| Epoch|Training Loss|Validation Loss|Accuracy |
| -----|-------------|---------------|--------- |
| 1 |No log |0.172788 |0.957 |
| 2 |0.194500 |0.202991 |0.956 |
| 3 |0.194500 |0.229950 |0.958 |
| 4 |0.038400 |0.267390 |0.954 |
| 5 |0.038400 |0.283116 |0.963 |
| 6 |0.007000 |0.254960 |0.961 |
| 7 |0.007000 |0.299375 |0.961 |
| 8 |0.007900 |0.276321 |0.966 |
| 9 |0.007900 |0.275304 |0.967 |
| 10 |0.002000 |0.271234 |0.967 |
| |
| |
| |
| ### Framework versions |
| |
| - PEFT 0.8.2 |
