File size: 4,255 Bytes
9880dcf 6f52b33 9880dcf 5a612ca 9880dcf | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 | ---
language:
- multilingual
- en
- pt
- fr
- es
- uk
- de
license: apache-2.0
base_model: distilbert-base-multilingual-cased
pipeline_tag: text-classification
tags:
- multi-label-classification
- customer-feedback
- churn
- cloud-gaming
---
# Multi-label classification of cloud-gaming churn feedback
Fine-tuned `distilbert-base-multilingual-cased` that tags the **technical issue** a user complains
about in the free-text comment left when cancelling a cloud-gaming subscription. A single comment
may describe several problems at once, so this is a **multi-label** task over 6 classes.
Comments are short (median 73 characters) and multilingual — no translation step is used.
👉 **Live demo:** [multilingual-game-feedback-classification.streamlit.app](https://multilingual-game-feedback-classification.streamlit.app)
## Labels
| label | meaning |
|---|---|
| `ping_latency` | network delay, high ping, input lag |
| `frames_drop` | low FPS, stuttering, unstable stream |
| `unable_launch` | the game or service does not start |
| `mouse_keyboard_headset` | peripherals: mouse, keyboard, headset, controller |
| `game_bug` | a defect inside the game itself |
| `failed_save` | progress is lost or not saved |
## Usage
The model outputs 6 independent sigmoid probabilities. **Do not use a 0.5 threshold** — per-class
thresholds were tuned on the validation set and are shipped in `inference_config.json`. They give
macro-F1 0.620 on the test set versus 0.581 with a plain 0.5 threshold.
```python
import json
import torch
from huggingface_hub import hf_hub_download
from transformers import AutoModelForSequenceClassification, AutoTokenizer
REPO = "dar1bi/cloud-gaming-feedback-multilabel"
tokenizer = AutoTokenizer.from_pretrained(REPO)
model = AutoModelForSequenceClassification.from_pretrained(REPO).eval()
config = json.load(open(hf_hub_download(REPO, "inference_config.json")))
text = "Constant micro stutter and very low fps in The Finals"
encoded = tokenizer(text, truncation=True, padding="max_length",
max_length=config["max_len"], return_tensors="pt")
with torch.no_grad():
probabilities = torch.sigmoid(model(**encoded).logits).numpy()[0]
labels = [label for label, probability, threshold
in zip(config["labels"], probabilities, config["thresholds"])
if probability >= threshold]
print(labels) # ['frames_drop']
```
## Training
- **Base model:** `distilbert-base-multilingual-cased`
- **Objective:** `BCEWithLogitsLoss` with `pos_weight` to compensate for class imbalance
- **Hyperparameters:** learning rate 8e-5 (grid over 2e-5 / 5e-5 / 8e-5), 5 epochs with best-epoch
selection by validation macro-F1, batch size 16, max sequence length 64
- **Split:** iterative-stratified multi-label split 70 / 15 / 15 — 2760 / 584 / 612 examples
- **Hardware:** Apple MPS, ~290 s per configuration
## Evaluation
Primary metric is **macro-F1** — it weights rare classes equally with frequent ones.
| split | macro-F1 | micro-F1 | macro ROC-AUC | mAP |
|---|---|---|---|---|
| validation | 0.681 | 0.717 | 0.828 | 0.727 |
| test | 0.620 | 0.678 | 0.817 | 0.685 |
Per-class F1 on the test set: `ping_latency` 0.78, `unable_launch` 0.70,
`mouse_keyboard_headset` 0.67, `frames_drop` 0.62, `game_bug` 0.49, `failed_save` 0.46.
The model was selected on the validation set among seven approaches, from TF-IDF with classical
classifiers to a zero-shot LLM. A most-frequent-class baseline scores 0.119 macro-F1.
## Limitations
- **Noisy labels.** Labels are self-reported by users in the cancellation form, so they are neither
consistent nor mutually exclusive. Part of what looks like model error is actually label noise.
- **Rare classes.** `game_bug` and `failed_save` are recognised in fewer than half of real cases —
they are infrequent in the data and semantically overlap with the other classes.
- **Very short texts.** Comments under 30 characters often carry too little signal.
- **Single split, single seed.** Differences of 0.01–0.02 macro-F1 are within noise.
## Training data
Internal user feedback of a cloud-gaming service, collected at subscription cancellation
(4 616 records, 3 956 after cleaning). The dataset is confidential and is **not published**.
|