Upload README.md with huggingface_hub

6f52b33 verified 10 days ago

4.26 kB

language:
  - multilingual
  - en
  - pt
  - fr
  - es
  - uk
  - de
license: apache-2.0
base_model: distilbert-base-multilingual-cased
pipeline_tag: text-classification
tags:
  - multi-label-classification
  - customer-feedback
  - churn
  - cloud-gaming

Multi-label classification of cloud-gaming churn feedback

Fine-tuned distilbert-base-multilingual-cased that tags the technical issue a user complains about in the free-text comment left when cancelling a cloud-gaming subscription. A single comment may describe several problems at once, so this is a multi-label task over 6 classes.

Comments are short (median 73 characters) and multilingual — no translation step is used.

👉 Live demo: multilingual-game-feedback-classification.streamlit.app

Labels

label	meaning
`ping_latency`	network delay, high ping, input lag
`frames_drop`	low FPS, stuttering, unstable stream
`unable_launch`	the game or service does not start
`mouse_keyboard_headset`	peripherals: mouse, keyboard, headset, controller
`game_bug`	a defect inside the game itself
`failed_save`	progress is lost or not saved

Usage

The model outputs 6 independent sigmoid probabilities. Do not use a 0.5 threshold — per-class thresholds were tuned on the validation set and are shipped in inference_config.json. They give macro-F1 0.620 on the test set versus 0.581 with a plain 0.5 threshold.

import json

import torch
from huggingface_hub import hf_hub_download
from transformers import AutoModelForSequenceClassification, AutoTokenizer

REPO = "dar1bi/cloud-gaming-feedback-multilabel"

tokenizer = AutoTokenizer.from_pretrained(REPO)
model = AutoModelForSequenceClassification.from_pretrained(REPO).eval()
config = json.load(open(hf_hub_download(REPO, "inference_config.json")))

text = "Constant micro stutter and very low fps in The Finals"
encoded = tokenizer(text, truncation=True, padding="max_length",
                    max_length=config["max_len"], return_tensors="pt")
with torch.no_grad():
    probabilities = torch.sigmoid(model(**encoded).logits).numpy()[0]

labels = [label for label, probability, threshold
          in zip(config["labels"], probabilities, config["thresholds"])
          if probability >= threshold]
print(labels)   # ['frames_drop']

Training

Base model: distilbert-base-multilingual-cased
Objective: BCEWithLogitsLoss with pos_weight to compensate for class imbalance
Hyperparameters: learning rate 8e-5 (grid over 2e-5 / 5e-5 / 8e-5), 5 epochs with best-epoch selection by validation macro-F1, batch size 16, max sequence length 64
Split: iterative-stratified multi-label split 70 / 15 / 15 — 2760 / 584 / 612 examples
Hardware: Apple MPS, ~290 s per configuration

Evaluation

Primary metric is macro-F1 — it weights rare classes equally with frequent ones.

split	macro-F1	micro-F1	macro ROC-AUC	mAP
validation	0.681	0.717	0.828	0.727
test	0.620	0.678	0.817	0.685

Per-class F1 on the test set: ping_latency 0.78, unable_launch 0.70, mouse_keyboard_headset 0.67, frames_drop 0.62, game_bug 0.49, failed_save 0.46.

The model was selected on the validation set among seven approaches, from TF-IDF with classical classifiers to a zero-shot LLM. A most-frequent-class baseline scores 0.119 macro-F1.

Limitations

Noisy labels. Labels are self-reported by users in the cancellation form, so they are neither consistent nor mutually exclusive. Part of what looks like model error is actually label noise.
Rare classes. game_bug and failed_save are recognised in fewer than half of real cases — they are infrequent in the data and semantically overlap with the other classes.
Very short texts. Comments under 30 characters often carry too little signal.
Single split, single seed. Differences of 0.01–0.02 macro-F1 are within noise.

Training data

Internal user feedback of a cloud-gaming service, collected at subscription cancellation (4 616 records, 3 956 after cleaning). The dataset is confidential and is not published.