--- library_name: peft base_model: roberta-base language: - en metrics: - accuracy pipeline_tag: text-classification --- # Model Card for Model ID This model classifies whether the text/email is spam or ham. ## Model Details This model is a fine-tuned version of roberta-base using LoRA specifically for a binary classification task containing emails (subject + message) and a label whether it is spam or ham. ### Model Description The base model for this fine-tuning is roberta-base, which is a transformer-based model pre-trained on a large corpus of English data in a self-supervised fashion. RoBERTa is an optimized version of BERT, designed to perform better on natural language understanding tasks. We have applied LoRA to adapt the original RoBERTa model to the specific nuances of our binary classification problem. LoRA introduces low-rank matrices that are trained during the fine-tuning process, enabling the model to learn task-specific adaptations without altering the pre-trained weights directly. - **Developed by:** Likhith231 - **Model type:** Text Classification - **Finetuned from model:** Roberta Base ### Training Data The model was finetuned on truncated version of SetFit/enron_spam dataset. The SetFit/enron_spam consists of 33716 rows. The truncated version consists of 1000 train samples and 1000 test samples with columns, text and label. ### Model Sources - **Repository:** https://huggingface.co/FacebookAI/roberta-base ## Training Details - pretrained model= Roberta-base - all params = 125,313,028 - trainable params= 665,858 - trainable% = 0.531355766137899 ### Parameters - weight_decay = 0.01 - lr = 1e-3 - batch_size = 4 - num_epochs = 10 ### Results Epoch|Training Loss|Validation Loss|Accuracy -----|-------------|---------------|--------- 1 |No log |0.172788 |0.957 2 |0.194500 |0.202991 |0.956 3 |0.194500 |0.229950 |0.958 4 |0.038400 |0.267390 |0.954 5 |0.038400 |0.283116 |0.963 6 |0.007000 |0.254960 |0.961 7 |0.007000 |0.299375 |0.961 8 |0.007900 |0.276321 |0.966 9 |0.007900 |0.275304 |0.967 10 |0.002000 |0.271234 |0.967 ### Framework versions - PEFT 0.8.2