FinTwitRoBERTa-large

FinTwitRoBERTa-large is a language model pre-trained on financial tweets. It continues the pre-training of cardiffnlp/twitter-roberta-large-2022-154m to capture the tickers, jargon and communication style of financial Twitter, making it a strong starting point for sentiment analysis and other financial NLP tasks on tweets. It is the successor of FinTwitBERT.

Sentiment Analysis

The FinTwitRoBERTa-sentiment model fine-tunes this model to classify financial tweets as bullish, bearish or neutral.

Fine-tuned with the same sentiment recipe, starting from FinTwitRoBERTa-large instead of cardiffnlp/twitter-roberta-large-2022-154m changes the macro F1 on our held-out test set of 1,237 financial tweets from 0.807 to 0.840 (mean over seeds).

Dataset

FinTwitRoBERTa-large is pre-trained on financial tweet datasets, consisting of tweets mentioning stocks and cryptocurrencies:

• StephanAkkerman/financial-tweets, shown 3 times per epoch to upweight it
• StephanAkkerman/crypto-stock-tweets

Before training, the tweets were cleaned: near-duplicates (such as bot price alerts that only differ in numbers), one-word tweets and tweets that are mostly hashtags, cashtags, mentions or links were removed, and quoted tweets became examples of their own. The tweets of our sentiment test and dev sets were removed from the pre-training data.

Model Details

• Base model: cardiffnlp/twitter-roberta-large-2022-154m, a RoBERTa-large model (355M parameters) trained on 154M tweets up to the end of 2022 (TimeLMs).
• Tokenizer: the tokenizer of the base model, unchanged. The mask token is <mask>.
• Preprocessing: links are replaced with [URL] and user mentions with @USER. These are not added as special tokens, so do the same replacement on your own tweets for the best results.
• Training: 1 epoch of masked language modeling (15% masking), learning rate 5e-05 with 6% warmup, weight decay 0.01, batch size 256, maximum length 128 tokens, on NVIDIA A100 80GB.
• Perplexity: 3.43 on 1,986 held-out financial tweets (our sentiment test and dev sets, removed from the pre-training data), against 166 for the base model. These tweets come from the same feed as part of the pre-training data, so expect a smaller gap on unrelated tweets; the sentiment results are the better measure.

More Information

For a comprehensive overview, including the complete training setup details and more, visit the ModernFinTwitBERT GitHub repository.

Usage

Using HuggingFace's transformers library the model and tokenizer can be converted into a pipeline for masked language modeling.

from transformers import pipeline

pipe = pipeline(
    "fill-mask",
    model="StephanAkkerman/FinTwitRoBERTa-large",
)
print(pipe("Fed cuts rates, $SPY to all time <mask>."))

Citing & Authors

If you use FinTwitRoBERTa-large in your research, please cite me as follows:

@misc{FinTwitRoBERTalarge,
  author = {Stephan Akkerman},
  title = {FinTwitRoBERTa-large},
  year = {2026},
  publisher = {GitHub},
  journal = {GitHub repository},
  howpublished = {\url{https://github.com/StephanAkkerman/ModernFinTwitBERT}}
}

License

This project is licensed under the MIT License. See the LICENSE file for details.