File size: 3,338 Bytes
bcb2f8a 0b001d0 bcb2f8a | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 | ---
license: cc-by-4.0
---
# Fine-Tuning BERT for Italian Public Communication Classification
This repository provides code and documentation for fine-tuning a BERT-based transformer model (`dbmdz/bert-base-italian-cased`) to classify Italian texts into public communication categories.
This research investigates how closely an automated text-classification pipeline can replicate—or even complement—human judgment. Specifically, we compare the category assignments that three expert researchers gave to a corpus of Italian Facebook posts with the predictions made by a model fine-tuned on the same human labels. To run this experiment, we started from dbmdz/bert-base-italian-cased and fine-tuned it on the researchers’ annotations, so that every subsequent prediction could be directly matched against each human coder and their inter-coder agreement scores.
---
## 🧠 Task: Multiclass Text Classification
Each input text is associated with **one and only one category** out of a predefined set (e.g., institutional communication, participatory communication, branding, etc.).
Unlike multilabel classification (where multiple categories may apply), this is a **multiclass classification task**.
---
## 📂 Dataset
The input data is expected in an Excel format (`.xlsx`) with at least the following columns:
- `Text`: the input text
- `ricercatoreX`: the annotated category (one per row)
Categories are label-encoded before training. The script uses `sklearn.preprocessing.LabelEncoder`.
---
## 🏗️ Model
- Base model: [`dbmdz/bert-base-italian-cased`](https://huggingface.co/dbmdz/bert-base-italian-cased)
- Classification head: dropout + dense layer (8 output classes)
- Loss function: `CrossEntropyLoss`
- Activation: `Softmax` at inference time
---
## 🚀 Training
To train the model:
```bash
pip install transformers torch scikit-learn pandas openpyxl
python bert_multiclass_classification.py
```
Training is done over 5 epochs with a batch size of 8 and learning rate of 1e-5.
The script:
- Loads and filters the dataset
- Encodes labels
- Prepares `CustomDataset` and PyTorch `DataLoader`
- Trains the model using BERT + dropout + classification layer
- Evaluates using accuracy and F1-score (micro, macro, weighted)
- Saves the model with `torch.save()`
---
## 🔍 Inference Example
Once trained, you can load the model and predict the class of a new text like this:
```python
text = "Aperte le iscrizioni"
input_data = prepare_input(text, tokenizer, max_len)
with torch.no_grad():
outputs = model(input_data['ids'], input_data['mask'], input_data['token_type_ids'])
predicted_class = torch.argmax(outputs, dim=1).cpu().numpy()[0]
predicted_label = label_encoder.inverse_transform([predicted_class])[0]
print(f"Predicted label: {predicted_label}")
```
---
## 📊 Evaluation
Evaluation metrics printed at each epoch:
- Accuracy
- F1-score (micro, macro, weighted)
Also includes a normalized confusion matrix using `seaborn`.
---
## 💾 Model Saving & Loading
The trained model is saved with:
```python
torch.save(model.state_dict(), "model_epoch_X.pth")
```
To reload:
```python
model.load_state_dict(torch.load("model_epoch_X.pth"))
```
---
## 🧑💻 Author
Developed by [anucita](https://huggingface.co/anucita) for research on public communication on social media.
|