| --- |
| license: cc-by-4.0 |
| --- |
| # Fine-Tuning BERT for Italian Public Communication Classification |
|
|
| This repository provides code and documentation for fine-tuning a BERT-based transformer model (`dbmdz/bert-base-italian-cased`) to classify Italian texts into public communication categories. |
| This research investigates how closely an automated text-classification pipeline can replicate—or even complement—human judgment. Specifically, we compare the category assignments that three expert researchers gave to a corpus of Italian Facebook posts with the predictions made by a model fine-tuned on the same human labels. To run this experiment, we started from dbmdz/bert-base-italian-cased and fine-tuned it on the researchers’ annotations, so that every subsequent prediction could be directly matched against each human coder and their inter-coder agreement scores. |
| --- |
|
|
| ## 🧠 Task: Multiclass Text Classification |
|
|
| Each input text is associated with **one and only one category** out of a predefined set (e.g., institutional communication, participatory communication, branding, etc.). |
|
|
| Unlike multilabel classification (where multiple categories may apply), this is a **multiclass classification task**. |
|
|
| --- |
|
|
| ## 📂 Dataset |
|
|
| The input data is expected in an Excel format (`.xlsx`) with at least the following columns: |
| - `Text`: the input text |
| - `ricercatoreX`: the annotated category (one per row) |
|
|
| Categories are label-encoded before training. The script uses `sklearn.preprocessing.LabelEncoder`. |
|
|
| --- |
|
|
| ## 🏗️ Model |
|
|
| - Base model: [`dbmdz/bert-base-italian-cased`](https://huggingface.co/dbmdz/bert-base-italian-cased) |
| - Classification head: dropout + dense layer (8 output classes) |
| - Loss function: `CrossEntropyLoss` |
| - Activation: `Softmax` at inference time |
|
|
| --- |
|
|
| ## 🚀 Training |
|
|
| To train the model: |
|
|
| ```bash |
| pip install transformers torch scikit-learn pandas openpyxl |
| python bert_multiclass_classification.py |
| ``` |
|
|
| Training is done over 5 epochs with a batch size of 8 and learning rate of 1e-5. |
|
|
| The script: |
| - Loads and filters the dataset |
| - Encodes labels |
| - Prepares `CustomDataset` and PyTorch `DataLoader` |
| - Trains the model using BERT + dropout + classification layer |
| - Evaluates using accuracy and F1-score (micro, macro, weighted) |
| - Saves the model with `torch.save()` |
|
|
| --- |
|
|
| ## 🔍 Inference Example |
|
|
| Once trained, you can load the model and predict the class of a new text like this: |
|
|
| ```python |
| text = "Aperte le iscrizioni" |
| input_data = prepare_input(text, tokenizer, max_len) |
| with torch.no_grad(): |
| outputs = model(input_data['ids'], input_data['mask'], input_data['token_type_ids']) |
| predicted_class = torch.argmax(outputs, dim=1).cpu().numpy()[0] |
| predicted_label = label_encoder.inverse_transform([predicted_class])[0] |
| print(f"Predicted label: {predicted_label}") |
| ``` |
|
|
| --- |
|
|
| ## 📊 Evaluation |
|
|
| Evaluation metrics printed at each epoch: |
| - Accuracy |
| - F1-score (micro, macro, weighted) |
|
|
| Also includes a normalized confusion matrix using `seaborn`. |
|
|
| --- |
|
|
| ## 💾 Model Saving & Loading |
|
|
| The trained model is saved with: |
|
|
| ```python |
| torch.save(model.state_dict(), "model_epoch_X.pth") |
| ``` |
|
|
| To reload: |
|
|
| ```python |
| model.load_state_dict(torch.load("model_epoch_X.pth")) |
| ``` |
|
|
| --- |
|
|
| ## 🧑💻 Author |
|
|
| Developed by [anucita](https://huggingface.co/anucita) for research on public communication on social media. |
|
|
|
|
