| | --- |
| | library_name: transformers |
| | license: apache-2.0 |
| | base_model: answerdotai/ModernBERT-base |
| | tags: |
| | - generated_from_trainer |
| | metrics: |
| | - accuracy |
| | - f1 |
| | - precision |
| | - recall |
| | model-index: |
| | - name: modernbert_fingpt_results |
| | results: [] |
| | datasets: |
| | - FinGPT/fingpt-sentiment-train |
| | --- |
| | |
| | # ModernBERT Fine-tuned for Financial Text Sentiment Analysis |
| |
|
| | This project fine-tunes the **ModernBERT** model on the **FinGPT** sentiment dataset for financial text sentiment analysis. |
| |
|
| | ## Dataset & Model |
| |
|
| | - **Model**: [answerdotai/ModernBERT-base](https://huggingface.co/answerdotai/ModernBERT-base) |
| | - **Dataset**: [FinGPT/fingpt-sentiment-train](https://huggingface.co/datasets/FinGPT/fingpt-sentiment-train) |
| | - **Task**: Multi-class sentiment classification (9 categories) |
| | - **Domain**: Financial text analysis |
| |
|
| | ### ModernBert |
| | ModernBERT is a modernized bidirectional encoder-only Transformer model (BERT-style) pre-trained on 2 trillion tokens of English and code data with a native context length of up to 8,192 tokens. |
| | It leverages architectural improvements such as Rotary Positional Embeddings (RoPE) for long-context support, Local-Global Alternating Attention for efficiency on long inputs, Unpadding and Flash Attention for efficient inference. |
| |
|
| | ### FinGPT Sentiment Analysis Dataset |
| | Contains 76,772 test rows (17,919,695 tokens) |
| |
|
| | ## Sentiment Categories |
| |
|
| | The model classifies text into 9 fine-grained sentiment levels: |
| |
|
| | | Label ID | Sentiment Category | Description | |
| | |----------|-------------------|-------------| |
| | | 0 | Strong Negative | Very pessimistic | |
| | | 1 | Moderately Negative | Somewhat pessimistic | |
| | | 2 | Mildly Negative | Slightly pessimistic | |
| | | 3 | Negative | General negative sentiment | |
| | | 4 | Neutral | No clear positive or negative bias | |
| | | 5 | Mildly Positive | Slightly optimistic | |
| | | 6 | Moderately Positive | Somewhat optimistic | |
| | | 7 | Positive | General positive sentiment | |
| | | 8 | Strong Positive | Very optimistic | |
| |
|
| | ## Model Configuration |
| |
|
| | ### Parameters |
| | - **Max Sequence Length**: 512 tokens |
| | - **Batch Size**: 16 |
| | - **Learning Rate**: 2e-5 with warmup |
| | - **Epochs**: 3 with early stopping |
| | - **Optimizer**: AdamW with weight decay (0.01) |
| |
|
| | ### Features |
| | - **Early Stopping**: Prevents overfitting (patience=3) |
| | - **Best Model Loading**: Automatically loads best checkpoint |
| | - **Mixed Precision**: FP16 training for speed optimization |
| | - **Stratified Splitting**: 80/20 train/validation split |
| |
|
| | ## Evaluation Metrics |
| | - **Accuracy**: Overall classification accuracy |
| | - **F1-Score**: Weighted F1-score across all classes |
| | - **Precision**: Weighted precision |
| | - **Recall**: Weighted recall |
| | - **Confusion Matrix**: Visual analysis of classification performance |
| | - **Classification Report**: Detailed per-class metrics |
| |
|
| | ## Performance |
| |
|
| | ### Training Time (on T4 GPU) |
| | - **Total Training**: ~30-45 minutes |
| | - **Per Epoch**: ~10-15 minutes |
| | - **Evaluation**: ~2-3 minutes |
| |
|
| | ### Training Results (Actual) |
| |
|
| | - Loss: 0.3741 |
| | - Accuracy: 0.9043 |
| | - F1: 0.9026 |
| | - Precision: 0.9022 |
| | - Recall: 0.9043 |
| |
|
| |
|
| | | Training Loss | Epoch | Step | Validation Loss | Accuracy | F1 | Precision | Recall | |
| | |:-------------:|:------:|:-----:|:---------------:|:--------:|:------:|:---------:|:------:| |
| | | 0.9551 | 0.1302 | 500 | 0.8504 | 0.6769 | 0.6623 | 0.6589 | 0.6769 | |
| | | 0.6639 | 0.2605 | 1000 | 0.7921 | 0.7162 | 0.6952 | 0.7444 | 0.7162 | |
| | | 0.5221 | 0.3907 | 1500 | 0.5066 | 0.8134 | 0.8083 | 0.8147 | 0.8134 | |
| | | 0.4415 | 0.5210 | 2000 | 0.4247 | 0.8381 | 0.8363 | 0.8410 | 0.8381 | |
| | | 0.4276 | 0.6512 | 2500 | 0.3884 | 0.8594 | 0.8486 | 0.8484 | 0.8594 | |
| | | 0.3767 | 0.7815 | 3000 | 0.3472 | 0.8756 | 0.8661 | 0.8689 | 0.8756 | |
| | | 0.3281 | 0.9117 | 3500 | 0.3463 | 0.8754 | 0.8631 | 0.8611 | 0.8754 | |
| | | 0.2419 | 1.0419 | 4000 | 0.3556 | 0.8883 | 0.8737 | 0.8728 | 0.8883 | |
| | | 0.2859 | 1.1722 | 4500 | 0.3162 | 0.8922 | 0.8859 | 0.8829 | 0.8922 | |
| | | 0.226 | 1.3024 | 5000 | 0.3269 | 0.8914 | 0.8857 | 0.8851 | 0.8914 | |
| | | 0.2378 | 1.4327 | 5500 | 0.3281 | 0.8903 | 0.8834 | 0.8881 | 0.8903 | |
| | | 0.2654 | 1.5629 | 6000 | 0.3038 | 0.8938 | 0.8862 | 0.8896 | 0.8938 | |
| | | 0.2319 | 1.6931 | 6500 | 0.3032 | 0.8993 | 0.8919 | 0.8905 | 0.8993 | |
| | | 0.2116 | 1.8234 | 7000 | 0.3013 | 0.9023 | 0.8919 | 0.8937 | 0.9023 | |
| | | 0.1922 | 1.9536 | 7500 | 0.2959 | 0.9017 | 0.8968 | 0.8941 | 0.9017 | |
| | | 0.1536 | 2.0839 | 8000 | 0.3983 | 0.9009 | 0.8986 | 0.9000 | 0.9009 | |
| | | 0.1438 | 2.2141 | 8500 | 0.3982 | 0.8990 | 0.8968 | 0.8954 | 0.8990 | |
| | | 0.1329 | 2.3444 | 9000 | 0.3809 | 0.9021 | 0.8990 | 0.8968 | 0.9021 | |
| | | 0.1175 | 2.4746 | 9500 | 0.3944 | 0.9019 | 0.8991 | 0.8977 | 0.9019 | |
| | | 0.1634 | 2.6048 | 10000 | 0.3899 | 0.9043 | 0.8999 | 0.8989 | 0.9043 | |
| | | 0.1049 | 2.7351 | 10500 | 0.4006 | 0.9037 | 0.9016 | 0.9009 | 0.9037 | |
| | | 0.1247 | 2.8653 | 11000 | 0.3828 | 0.9053 | 0.9019 | 0.9006 | 0.9053 | |
| | | 0.1511 | 2.9956 | 11500 | 0.3741 | 0.9043 | 0.9026 | 0.9022 | 0.9043 | |
| |
|
| | ## Deployment Options |
| | - **API Deployment**: Create REST API using FastAPI |
| | - **Batch Processing**: Set up automated sentiment analysis pipeline |
| | - **Real-time Analysis**: Integrate with financial data streams |
| |
|
| | ## References |
| | - [ModernBERT Paper](https://arxiv.org/abs/2412.13663) |
| | - [FinGPT Project](https://github.com/AI4Finance-Foundation/FinGPT) |
| | - [Hugging Face Transformers](https://huggingface.co/docs/transformers) |
| | - [Financial Sentiment Analysis Survey](https://arxiv.org/abs/2212.14197) |
