README.md

---
language:
- multilingual
- en
- pt
- fr
- es
- uk
- de
license: apache-2.0
base_model: distilbert-base-multilingual-cased
pipeline_tag: text-classification
tags:
- multi-label-classification
- customer-feedback
- churn
- cloud-gaming
---

# Multi-label classification of cloud-gaming churn feedback

Fine-tuned `distilbert-base-multilingual-cased` that tags the **technical issue** a user complains
about in the free-text comment left when cancelling a cloud-gaming subscription. A single comment
may describe several problems at once, so this is a **multi-label** task over 6 classes.

Comments are short (median 73 characters) and multilingual — no translation step is used.

👉 **Live demo:** [multilingual-game-feedback-classification.streamlit.app](https://multilingual-game-feedback-classification.streamlit.app)

## Labels

| label | meaning |
|---|---|
| `ping_latency` | network delay, high ping, input lag |
| `frames_drop` | low FPS, stuttering, unstable stream |
| `unable_launch` | the game or service does not start |
| `mouse_keyboard_headset` | peripherals: mouse, keyboard, headset, controller |
| `game_bug` | a defect inside the game itself |
| `failed_save` | progress is lost or not saved |

## Usage

The model outputs 6 independent sigmoid probabilities. **Do not use a 0.5 threshold** — per-class
thresholds were tuned on the validation set and are shipped in `inference_config.json`. They give
macro-F1 0.620 on the test set versus 0.581 with a plain 0.5 threshold.

```python
import json

import torch
from huggingface_hub import hf_hub_download
from transformers import AutoModelForSequenceClassification, AutoTokenizer

REPO = "dar1bi/cloud-gaming-feedback-multilabel"

tokenizer = AutoTokenizer.from_pretrained(REPO)
model = AutoModelForSequenceClassification.from_pretrained(REPO).eval()
config = json.load(open(hf_hub_download(REPO, "inference_config.json")))

text = "Constant micro stutter and very low fps in The Finals"
encoded = tokenizer(text, truncation=True, padding="max_length",
                    max_length=config["max_len"], return_tensors="pt")
with torch.no_grad():
    probabilities = torch.sigmoid(model(**encoded).logits).numpy()[0]

labels = [label for label, probability, threshold
          in zip(config["labels"], probabilities, config["thresholds"])
          if probability >= threshold]
print(labels)   # ['frames_drop']
```

## Training

- **Base model:** `distilbert-base-multilingual-cased`
- **Objective:** `BCEWithLogitsLoss` with `pos_weight` to compensate for class imbalance
- **Hyperparameters:** learning rate 8e-5 (grid over 2e-5 / 5e-5 / 8e-5), 5 epochs with best-epoch
  selection by validation macro-F1, batch size 16, max sequence length 64
- **Split:** iterative-stratified multi-label split 70 / 15 / 15 — 2760 / 584 / 612 examples
- **Hardware:** Apple MPS, ~290 s per configuration

## Evaluation

Primary metric is **macro-F1** — it weights rare classes equally with frequent ones.

| split | macro-F1 | micro-F1 | macro ROC-AUC | mAP |
|---|---|---|---|---|
| validation | 0.681 | 0.717 | 0.828 | 0.727 |
| test | 0.620 | 0.678 | 0.817 | 0.685 |

Per-class F1 on the test set: `ping_latency` 0.78, `unable_launch` 0.70,
`mouse_keyboard_headset` 0.67, `frames_drop` 0.62, `game_bug` 0.49, `failed_save` 0.46.

The model was selected on the validation set among seven approaches, from TF-IDF with classical
classifiers to a zero-shot LLM. A most-frequent-class baseline scores 0.119 macro-F1.

## Limitations

- **Noisy labels.** Labels are self-reported by users in the cancellation form, so they are neither
  consistent nor mutually exclusive. Part of what looks like model error is actually label noise.
- **Rare classes.** `game_bug` and `failed_save` are recognised in fewer than half of real cases —
  they are infrequent in the data and semantically overlap with the other classes.
- **Very short texts.** Comments under 30 characters often carry too little signal.
- **Single split, single seed.** Differences of 0.01–0.02 macro-F1 are within noise.

## Training data

Internal user feedback of a cloud-gaming service, collected at subscription cancellation
(4 616 records, 3 956 after cleaning). The dataset is confidential and is **not published**.