FinTwitRoBERTa-sentiment

FinTwitRoBERTa-sentiment is a fine-tuned model for classifying the sentiment of financial tweets as bullish, bearish or neutral. It uses FinTwitRoBERTa-large as a base model, a RoBERTa-large model (355M parameters) pre-trained on millions of financial tweets. This approach ensures that FinTwitRoBERTa-sentiment has seen enough financial tweets, which have an informal nature, compared to other financial texts, such as news headlines. Therefore this model performs great on informal financial texts, seen on social media. It is the successor of FinTwitBERT-sentiment.

Intended Uses

FinTwitRoBERTa-sentiment is intended for classifying financial tweets or other financial social media texts, such as a feed of tweets about stocks and crypto. A tweet is bullish or bearish only when its author expresses an expectation or position; facts, news, questions and price alerts are neutral, which is most of a typical feed. The sentiment is that of the tweet's own text: for a quote tweet, classify the new text without the quoted tweet.

Dataset

FinTwitRoBERTa-sentiment has been trained on human-labeled financial tweets from two datasets:

• TimKoornstra/financial-tweets-sentiment: 24,373 human-labeled tweets after cleaning. The sources without a neutral class or with inconsistent labels were left out, and the bullish and bearish tweets of the other sources were relabeled by hand with one labeling guideline (most became neutral, since they state facts or news rather than an opinion).
• 1,500 tweets from StephanAkkerman/financial-tweets, posted from 2024 onwards, picked where an earlier model was least sure and labeled with the same guideline (data/train/feed_v1.csv)

In total 23,435 tweets for training and 2,438 for validation. The tweets of our dev and test sets were removed from the training data.

Model Details

• Base model: FinTwitRoBERTa-large, a RoBERTa-large model (355M parameters) pre-trained on millions of financial tweets.
• Labels: NEUTRAL, BULLISH, BEARISH.
• Training: 2 epochs, learning rate 2e-05 with 10% warmup and layer-wise learning-rate decay of 0.9, weight decay 0.01, batch size 32, maximum length 128 tokens, seed 42. The checkpoint with the best validation F1 (2,438 tweets held out from the training sources) is kept. The recipe (llrd-0.9-feed in experiments.json) was chosen out of a grid of learning rates, epochs, class weights, label smoothing and layer-wise decay by its macro F1 on the dev set.
• Calibration: config.calibration_temperature = 1.046, fitted on the dev set. Divide the logits by it before the softmax to get confidence scores that match how often the model is right.

Evaluation

The test set has 1,237 tweets from a real financial Twitter feed (first seen from August 2023 onwards) and the dev set 749 tweets from the same feed (2024 onwards), labeled with the same guideline. Neither was used for training; the dev set was used to choose the recipe and fit the temperature. The 95% bootstrap interval of the test macro F1 is 0.871–0.910. For comparison, FinTwitBERT-sentiment scores a macro F1 of 0.565 on the same test set.

Metric	Test (1,237 tweets)	Dev (749 tweets)
Macro F1	0.891	0.826
Accuracy	0.906	0.887
Bullish precision	0.884	0.789
Bullish recall	0.903	0.783
Bearish precision	0.938	0.836
Bearish recall	0.787	0.708
Neutral recall	0.934	0.935
Calibration error (ECE)	0.017	0.033

More Information

For a comprehensive overview, including the training setup and analysis of the model, visit the ModernFinTwitBERT GitHub repository.

Usage

Using HuggingFace's transformers library the model and tokenizers can be converted into a pipeline for text classification.

from transformers import pipeline

# Create a sentiment analysis pipeline
pipe = pipeline(
    "sentiment-analysis",
    model="StephanAkkerman/FinTwitRoBERTa-sentiment",
)

# Get the predicted sentiment
print(pipe("Nice 9% pre market move for $para, pump my calls Uncle Buffett 🤑"))

The model was trained on tweets with links replaced by [URL] and user mentions by @USER, so do the same for the best results. The pipeline's scores are not calibrated; for confidence scores, divide the logits by config.calibration_temperature:

import torch
from transformers import AutoModelForSequenceClassification, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("StephanAkkerman/FinTwitRoBERTa-sentiment")
model = AutoModelForSequenceClassification.from_pretrained("StephanAkkerman/FinTwitRoBERTa-sentiment").eval()

inputs = tokenizer(["$NVDA breaking out, loading calls"], return_tensors="pt", truncation=True)
with torch.no_grad():
    logits = model(**inputs).logits
probs = (logits / model.config.calibration_temperature).softmax(-1)
print({model.config.id2label[i]: p for i, p in enumerate(probs[0].tolist())})

Decision thresholds

config.decision_thresholds = {"bullish": 0.8, "bearish": 0.86}, tuned on the dev set for precision >= 0.9. Call a tweet bullish or bearish only when its calibrated probability for that class reaches the threshold, otherwise neutral:

label = model.config.id2label[int(probs[0].argmax())].lower()
if label != "neutral" and probs[0].max() < model.config.decision_thresholds[label]:
    label = "neutral"

split	setting	macro_f1	bullish_calls	bullish_precision	bullish_recall	bearish_calls	bearish_precision	bearish_recall
dev	argmax	0.826	142	0.789	0.783	55	0.836	0.708
dev	thresholds	0.763	102	0.912	0.65	33	0.909	0.462
test	argmax	0.891	388	0.884	0.903	130	0.938	0.787
test	thresholds	0.851	325	0.954	0.816	98	0.99	0.626

ONNX int8

onnx/model_int8.onnx is this model with int8 weights (dynamic per-channel quantization of all linear layers and the embeddings) for fast CPU inference, for example on a Raspberry Pi. It needs only onnxruntime and tokenizers, no torch:

	fp32 (PyTorch)	int8 (ONNX)
size	1,421 MB	358 MB
ms per tweet (batch 1, 8 CPU threads)	97	19

import json

import numpy as np
import onnxruntime as ort
from huggingface_hub import hf_hub_download
from tokenizers import Tokenizer

repo = "StephanAkkerman/FinTwitRoBERTa-sentiment"
config = json.loads(open(hf_hub_download(repo, "config.json")).read())
session = ort.InferenceSession(hf_hub_download(repo, "onnx/model_int8.onnx"))
tokenizer = Tokenizer.from_file(hf_hub_download(repo, "tokenizer.json"))
tokenizer.enable_truncation(128)
tokenizer.enable_padding(pad_id=1)

encodings = tokenizer.encode_batch(["$NVDA breaking out, loading calls"])
logits = session.run(None, {
    "input_ids": np.array([e.ids for e in encodings], dtype=np.int64),
    "attention_mask": np.array([e.attention_mask for e in encodings], dtype=np.int64),
})[0] / config["calibration_temperature"]
probs = np.exp(logits) / np.exp(logits).sum(-1, keepdims=True)  # NEUTRAL, BULLISH, BEARISH

Apply config.decision_thresholds to probs the same way as for the PyTorch model. ModernFinTwitBERT's onnx_predict.OnnxSentiment does all of this, including the training preprocessing (@USER, [URL]). Scores on the same dev and test sets; same_label_as_fp32 is how often both models give the same argmax label:

split	model	setting	macro_f1	bullish_calls	bullish_precision	bullish_recall	bearish_calls	bearish_precision	bearish_recall	same_label_as_fp32
dev	fp32	argmax	0.826	142	0.789	0.783	55	0.836	0.708	0.979
dev	int8	argmax	0.818	139	0.799	0.776	56	0.804	0.692	0.979
dev	fp32	thresholds	0.763	102	0.912	0.65	33	0.909	0.462	0.979
dev	int8	thresholds	0.76	98	0.918	0.629	32	0.938	0.462	0.979
test	fp32	argmax	0.891	388	0.884	0.903	130	0.938	0.787	0.989
test	int8	argmax	0.892	383	0.896	0.903	133	0.925	0.794	0.989
test	fp32	thresholds	0.851	325	0.954	0.816	98	0.99	0.626	0.989
test	int8	thresholds	0.844	321	0.956	0.808	95	0.989	0.606	0.989

Citing & Authors

If you use FinTwitRoBERTa-sentiment or FinTwitRoBERTa-large in your research, please cite me as follows:

@misc{FinTwitRoBERTalarge,
  author = {Stephan Akkerman},
  title = {FinTwitRoBERTa-large},
  year = {2026},
  publisher = {GitHub},
  journal = {GitHub repository},
  howpublished = {\url{https://github.com/StephanAkkerman/ModernFinTwitBERT}}
}

Additionally, if you utilize the sentiment classifier, please cite:

@misc{FinTwitRoBERTasentiment,
  author = {Stephan Akkerman},
  title = {FinTwitRoBERTa-sentiment: A Sentiment Classifier for Financial Tweets},
  year = {2026},
  publisher = {Hugging Face},
  howpublished = {\url{https://huggingface.co/StephanAkkerman/FinTwitRoBERTa-sentiment}}
}

License

This project is licensed under the MIT License. See the LICENSE file for details.