File size: 5,737 Bytes
6aafa44 1950653 76747cd 6aafa44 76747cd 1950653 76747cd 1950653 4ec4bd6 76747cd 1950653 76747cd | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 | ---
license: cc-by-4.0
arxiv: 2508.16873
library_name: pytorch
pipeline_tag: text-classification
tags:
- sentiment-analysis
- multimodal
- image-sentiment
- perceptsent
- mllm
language:
- en
base_model:
- answerdotai/ModernBERT-large
- facebook/bart-large-mnli
---
# MLLMsent — sentiment classifiers over multimodal-LLM image descriptions
Fine-tuned text classifiers from **"Multimodal LLMs See Sentiment"**
([arXiv:2508.16873](https://arxiv.org/abs/2508.16873)). Each checkpoint scores the sentiment of an image
*description* produced by a multimodal LLM, which is the second stage of the MLLMsent
pipeline.
- **Paper:** [arXiv:2508.16873](https://arxiv.org/abs/2508.16873)
- **Code, training and inference:** https://github.com/neemiasbsilva/multimodal-LLMs-see-sentiment
- **Datasets (inputs, captions and every result CSV):** https://huggingface.co/datasets/Neemias/multimodal-LLMs-See-Sentiment
## Pipeline
```
image ──▶ multimodal LLM ──▶ description ──▶ text classifier ──▶ sentiment
(GPT-4o mini, Gemini, (these checkpoints:
DeepSeek-VL2, Phi-4, ModernBERT-large,
Gemma-4, MiniGPT-4) BART-large-MNLI)
```
The paper's best configuration is **GPT-4o mini captions + fine-tuned ModernBERT**.
## Layout
```
{caption_mllm}/{backbone}/{problem}/sigma{n}/{finetuned|not_finetuned}/
model.safetensors
config.json
MANIFEST.json
```
- **problem** — label granularity: `p5` (5 classes), `p3` (3), `p2plus`/`p2neg` (2).
- **sigma** — annotator-agreement threshold used to filter the training set (3 or 5).
- **finetuned** — whole backbone trained. **not_finetuned** — backbone frozen, head only.
Every `config.json` carries the base model id, `id2label`/`label2id`, the source
checkpoint's SHA-256 and the 5-fold scores that checkpoint achieved.
## Coverage
| caption MLLM | base model | checkpoints |
|---|---|---|
| `deepseek` | `answerdotai/ModernBERT-large` | 8 |
| `deepseek` | `facebook/bart-large-mnli` | 6 |
| `gemini` | `answerdotai/ModernBERT-large` | 8 |
| `gemma4` | `answerdotai/ModernBERT-large` | 8 |
| `minigpt4` | `answerdotai/ModernBERT-large` | 8 |
| `minigpt4` | `facebook/bart-large-mnli` | 6 |
| `openai` | `answerdotai/ModernBERT-large` | 12 |
| `openai` | `facebook/bart-large-mnli` | 8 |
| `phi4` | `answerdotai/ModernBERT-large` | 8 |
Weights are **fp16 safetensors** converted from the original fp32 training checkpoints.
## Best checkpoints
| checkpoint | track | mean 5-fold F1 | classes |
|---|---|---|---|
| openai-modernbert-p3-sigma5 | finetuning | 0.9581 | 3 |
| openai-bart-p3-sigma5 | finetuning | 0.9532 | 3 |
| gemini-modernbert-p3-sigma5 | finetuning | 0.9451 | 3 |
| gemma4-modernbert-p3-sigma5 | finetuning | 0.9417 | 3 |
| phi4-modernbert-p3-sigma5 | finetuning | 0.9332 | 3 |
| openai-modernbert-p3-sigma5 | not-finetuning | 0.9063 | 3 |
| minigpt4-modernbert-p3-sigma5 | finetuning | 0.9039 | 3 |
| deepseek-modernbert-p3-sigma5 | finetuning | 0.8948 | 3 |
| openai-bart-p3-sigma5 | not-finetuning | 0.8543 | 3 |
| openai-modernbert-p5-sigma5 | finetuning | 0.8445 | 5 |
## Usage
```python
import json, torch
from huggingface_hub import hf_hub_download
from safetensors.torch import load_file
from transformers import AutoModel, AutoTokenizer
repo = "Neemias/multimodal-LLMs-See-Sentiment"
folder = "gpt4-openai-classify/modernbert/p3/sigma5/finetuned"
weights = load_file(hf_hub_download(repo, f"{folder}/model.safetensors"))
config = json.load(open(hf_hub_download(repo, f"{folder}/config.json")))
for alias, owner in config["tied_weights"].items():
weights[alias] = weights[owner]
class SentimentClassifier(torch.nn.Module):
def __init__(self, base_model, num_classes):
super().__init__()
self.model = AutoModel.from_pretrained(base_model)
self.classifier = torch.nn.Sequential(
torch.nn.Linear(self.model.config.hidden_size, 1024),
torch.nn.ReLU(),
torch.nn.Linear(1024, num_classes),
)
def forward(self, ids, mask):
return self.classifier(self.model(ids, attention_mask=mask).last_hidden_state[:, 0])
model = SentimentClassifier(config["base_model"], config["num_classes"])
model.load_state_dict({k: v.float() for k, v in weights.items()})
model.eval()
tokenizer = AutoTokenizer.from_pretrained(config["base_model"])
batch = tokenizer(["A bright park full of children playing."], return_tensors="pt",
padding="max_length", truncation=True, max_length=config["max_len"])
prediction = model(batch["input_ids"], batch["attention_mask"]).argmax(-1).item()
print(config["id2label"][str(prediction)])
```
Or through the project CLI:
```bash
mllmsent hub pull-checkpoint openai-modernbert-p3-sigma5
mllmsent predict --spec openai-modernbert-p3-sigma5 --input captions.csv --output predictions.csv
```
## Not published here
- **LLaMA-3 qLoRA adapters** — the adapter weights were never retained; only the
training logs and `adapter_config.json` survive.
- **Swin Transformer baseline** — its checkpoint-saving path was broken, so no
weights were ever written. Results for it are in the dataset repo.
- A few BART sigma-5 fine-tuned cells, for the same reason.
## Citation
```bibtex
@misc{dasilva2026multimodalllmssentiment,
title={Multimodal LLMs See Sentiment},
author={Neemias B. da Silva and John Harrison and Rodrigo Minetto and Myriam R. Delgado and Bogdan T. Nassu and Thiago H. Silva},
year={2026},
eprint={2508.16873},
archivePrefix={arXiv},
primaryClass={cs.CV},
url={https://arxiv.org/abs/2508.16873},
}
```
## License
CC-BY-4.0. The base models keep their own licenses.
|