File size: 5,821 Bytes
2af3074 133fc01 2af3074 a9e5d0a 7fe9e33 a9e5d0a 514217b a9e5d0a c64106e a9e5d0a c64106e a9e5d0a c64106e a9e5d0a 2af3074 6a2c820 2af3074 a9e5d0a 514217b | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 | ---
library_name: transformers
license: apache-2.0
base_model: answerdotai/ModernBERT-base
tags:
- generated_from_trainer
metrics:
- accuracy
- f1
- precision
- recall
model-index:
- name: modernbert_fingpt_results
results: []
datasets:
- FinGPT/fingpt-sentiment-train
---
# ModernBERT Fine-tuned for Financial Text Sentiment Analysis
This project fine-tunes the **ModernBERT** model on the **FinGPT** sentiment dataset for financial text sentiment analysis.
## Dataset & Model
- **Model**: [answerdotai/ModernBERT-base](https://huggingface.co/answerdotai/ModernBERT-base)
- **Dataset**: [FinGPT/fingpt-sentiment-train](https://huggingface.co/datasets/FinGPT/fingpt-sentiment-train)
- **Task**: Multi-class sentiment classification (9 categories)
- **Domain**: Financial text analysis
### ModernBert
ModernBERT is a modernized bidirectional encoder-only Transformer model (BERT-style) pre-trained on 2 trillion tokens of English and code data with a native context length of up to 8,192 tokens.
It leverages architectural improvements such as Rotary Positional Embeddings (RoPE) for long-context support, Local-Global Alternating Attention for efficiency on long inputs, Unpadding and Flash Attention for efficient inference.
### FinGPT Sentiment Analysis Dataset
Contains 76,772 test rows (17,919,695 tokens)
## Sentiment Categories
The model classifies text into 9 fine-grained sentiment levels:
| Label ID | Sentiment Category | Description |
|----------|-------------------|-------------|
| 0 | Strong Negative | Very pessimistic |
| 1 | Moderately Negative | Somewhat pessimistic |
| 2 | Mildly Negative | Slightly pessimistic |
| 3 | Negative | General negative sentiment |
| 4 | Neutral | No clear positive or negative bias |
| 5 | Mildly Positive | Slightly optimistic |
| 6 | Moderately Positive | Somewhat optimistic |
| 7 | Positive | General positive sentiment |
| 8 | Strong Positive | Very optimistic |
## Model Configuration
### Parameters
- **Max Sequence Length**: 512 tokens
- **Batch Size**: 16
- **Learning Rate**: 2e-5 with warmup
- **Epochs**: 3 with early stopping
- **Optimizer**: AdamW with weight decay (0.01)
### Features
- **Early Stopping**: Prevents overfitting (patience=3)
- **Best Model Loading**: Automatically loads best checkpoint
- **Mixed Precision**: FP16 training for speed optimization
- **Stratified Splitting**: 80/20 train/validation split
## Evaluation Metrics
- **Accuracy**: Overall classification accuracy
- **F1-Score**: Weighted F1-score across all classes
- **Precision**: Weighted precision
- **Recall**: Weighted recall
- **Confusion Matrix**: Visual analysis of classification performance
- **Classification Report**: Detailed per-class metrics
## Performance
### Training Time (on T4 GPU)
- **Total Training**: ~30-45 minutes
- **Per Epoch**: ~10-15 minutes
- **Evaluation**: ~2-3 minutes
### Training Results (Actual)
- Loss: 0.3741
- Accuracy: 0.9043
- F1: 0.9026
- Precision: 0.9022
- Recall: 0.9043
| Training Loss | Epoch | Step | Validation Loss | Accuracy | F1 | Precision | Recall |
|:-------------:|:------:|:-----:|:---------------:|:--------:|:------:|:---------:|:------:|
| 0.9551 | 0.1302 | 500 | 0.8504 | 0.6769 | 0.6623 | 0.6589 | 0.6769 |
| 0.6639 | 0.2605 | 1000 | 0.7921 | 0.7162 | 0.6952 | 0.7444 | 0.7162 |
| 0.5221 | 0.3907 | 1500 | 0.5066 | 0.8134 | 0.8083 | 0.8147 | 0.8134 |
| 0.4415 | 0.5210 | 2000 | 0.4247 | 0.8381 | 0.8363 | 0.8410 | 0.8381 |
| 0.4276 | 0.6512 | 2500 | 0.3884 | 0.8594 | 0.8486 | 0.8484 | 0.8594 |
| 0.3767 | 0.7815 | 3000 | 0.3472 | 0.8756 | 0.8661 | 0.8689 | 0.8756 |
| 0.3281 | 0.9117 | 3500 | 0.3463 | 0.8754 | 0.8631 | 0.8611 | 0.8754 |
| 0.2419 | 1.0419 | 4000 | 0.3556 | 0.8883 | 0.8737 | 0.8728 | 0.8883 |
| 0.2859 | 1.1722 | 4500 | 0.3162 | 0.8922 | 0.8859 | 0.8829 | 0.8922 |
| 0.226 | 1.3024 | 5000 | 0.3269 | 0.8914 | 0.8857 | 0.8851 | 0.8914 |
| 0.2378 | 1.4327 | 5500 | 0.3281 | 0.8903 | 0.8834 | 0.8881 | 0.8903 |
| 0.2654 | 1.5629 | 6000 | 0.3038 | 0.8938 | 0.8862 | 0.8896 | 0.8938 |
| 0.2319 | 1.6931 | 6500 | 0.3032 | 0.8993 | 0.8919 | 0.8905 | 0.8993 |
| 0.2116 | 1.8234 | 7000 | 0.3013 | 0.9023 | 0.8919 | 0.8937 | 0.9023 |
| 0.1922 | 1.9536 | 7500 | 0.2959 | 0.9017 | 0.8968 | 0.8941 | 0.9017 |
| 0.1536 | 2.0839 | 8000 | 0.3983 | 0.9009 | 0.8986 | 0.9000 | 0.9009 |
| 0.1438 | 2.2141 | 8500 | 0.3982 | 0.8990 | 0.8968 | 0.8954 | 0.8990 |
| 0.1329 | 2.3444 | 9000 | 0.3809 | 0.9021 | 0.8990 | 0.8968 | 0.9021 |
| 0.1175 | 2.4746 | 9500 | 0.3944 | 0.9019 | 0.8991 | 0.8977 | 0.9019 |
| 0.1634 | 2.6048 | 10000 | 0.3899 | 0.9043 | 0.8999 | 0.8989 | 0.9043 |
| 0.1049 | 2.7351 | 10500 | 0.4006 | 0.9037 | 0.9016 | 0.9009 | 0.9037 |
| 0.1247 | 2.8653 | 11000 | 0.3828 | 0.9053 | 0.9019 | 0.9006 | 0.9053 |
| 0.1511 | 2.9956 | 11500 | 0.3741 | 0.9043 | 0.9026 | 0.9022 | 0.9043 |
## Deployment Options
- **API Deployment**: Create REST API using FastAPI
- **Batch Processing**: Set up automated sentiment analysis pipeline
- **Real-time Analysis**: Integrate with financial data streams
## References
- [ModernBERT Paper](https://arxiv.org/abs/2412.13663)
- [FinGPT Project](https://github.com/AI4Finance-Foundation/FinGPT)
- [Hugging Face Transformers](https://huggingface.co/docs/transformers)
- [Financial Sentiment Analysis Survey](https://arxiv.org/abs/2212.14197) |