---
license: cc-by-4.0
---
# Fine-Tuning BERT for Italian Public Communication Classification

This repository provides code and documentation for fine-tuning a BERT-based transformer model (`dbmdz/bert-base-italian-cased`) to classify Italian texts into public communication categories.
This research investigates how closely an automated text-classification pipeline can replicate—or even complement—human judgment. Specifically, we compare the category assignments that three expert researchers gave to a corpus of Italian Facebook posts with the predictions made by a model fine-tuned on the same human labels. To run this experiment, we started from dbmdz/bert-base-italian-cased and fine-tuned it on the researchers’ annotations, so that every subsequent prediction could be directly matched against each human coder and their inter-coder agreement scores.
---

## 🧠 Task: Multiclass Text Classification

Each input text is associated with **one and only one category** out of a predefined set (e.g., institutional communication, participatory communication, branding, etc.).

Unlike multilabel classification (where multiple categories may apply), this is a **multiclass classification task**.

---

## 📂 Dataset

The input data is expected in an Excel format (`.xlsx`) with at least the following columns:
- `Text`: the input text
- `ricercatoreX`: the annotated category (one per row)

Categories are label-encoded before training. The script uses `sklearn.preprocessing.LabelEncoder`.

---

## 🏗️ Model

- Base model: [`dbmdz/bert-base-italian-cased`](https://huggingface.co/dbmdz/bert-base-italian-cased)
- Classification head: dropout + dense layer (8 output classes)
- Loss function: `CrossEntropyLoss`
- Activation: `Softmax` at inference time

---

## 🚀 Training

To train the model:

```bash
pip install transformers torch scikit-learn pandas openpyxl
python bert_multiclass_classification.py
```

Training is done over 5 epochs with a batch size of 8 and learning rate of 1e-5.

The script:
- Loads and filters the dataset
- Encodes labels
- Prepares `CustomDataset` and PyTorch `DataLoader`
- Trains the model using BERT + dropout + classification layer
- Evaluates using accuracy and F1-score (micro, macro, weighted)
- Saves the model with `torch.save()`

---

## 🔍 Inference Example

Once trained, you can load the model and predict the class of a new text like this:

```python
text = "Aperte le iscrizioni"
input_data = prepare_input(text, tokenizer, max_len)
with torch.no_grad():
    outputs = model(input_data['ids'], input_data['mask'], input_data['token_type_ids'])
predicted_class = torch.argmax(outputs, dim=1).cpu().numpy()[0]
predicted_label = label_encoder.inverse_transform([predicted_class])[0]
print(f"Predicted label: {predicted_label}")
```

---

## 📊 Evaluation

Evaluation metrics printed at each epoch:
- Accuracy
- F1-score (micro, macro, weighted)

Also includes a normalized confusion matrix using `seaborn`.

---

## 💾 Model Saving & Loading

The trained model is saved with:

```python
torch.save(model.state_dict(), "model_epoch_X.pth")
```

To reload:

```python
model.load_state_dict(torch.load("model_epoch_X.pth"))
```

---

## 🧑‍💻 Author

Developed by [anucita](https://huggingface.co/anucita) for research on public communication on social media.