Add fine-tuned DistilBERT with per-class thresholds

README.md

@@ -0,0 +1,106 @@
+---
+language:
+- multilingual
+- en
+- pt
+- fr
+- es
+- uk
+- de
+license: apache-2.0
+base_model: distilbert-base-multilingual-cased
+pipeline_tag: text-classification
+tags:
+- multi-label-classification
+- customer-feedback
+- churn
+- cloud-gaming
+---
+
+# Multi-label classification of cloud-gaming churn feedback
+
+Fine-tuned `distilbert-base-multilingual-cased` that tags the **technical issue** a user complains
+about in the free-text comment left when cancelling a cloud-gaming subscription. A single comment
+may describe several problems at once, so this is a **multi-label** task over 6 classes.
+
+Comments are short (median 73 characters) and multilingual — no translation step is used.
+
+## Labels
+
+| label | meaning |
+|---|---|
+| `ping_latency` | network delay, high ping, input lag |
+| `frames_drop` | low FPS, stuttering, unstable stream |
+| `unable_launch` | the game or service does not start |
+| `mouse_keyboard_headset` | peripherals: mouse, keyboard, headset, controller |
+| `game_bug` | a defect inside the game itself |
+| `failed_save` | progress is lost or not saved |
+
+## Usage
+
+The model outputs 6 independent sigmoid probabilities. **Do not use a 0.5 threshold** — per-class
+thresholds were tuned on the validation set and are shipped in `inference_config.json`. They give
+macro-F1 0.620 on the test set versus 0.581 with a plain 0.5 threshold.
+
+```python
+import json
+
+import torch
+from huggingface_hub import hf_hub_download
+from transformers import AutoModelForSequenceClassification, AutoTokenizer
+
+REPO = "<your-username>/<your-model-name>"
+
+tokenizer = AutoTokenizer.from_pretrained(REPO)
+model = AutoModelForSequenceClassification.from_pretrained(REPO).eval()
+config = json.load(open(hf_hub_download(REPO, "inference_config.json")))
+
+text = "Constant micro stutter and very low fps in The Finals"
+encoded = tokenizer(text, truncation=True, padding="max_length",
+                    max_length=config["max_len"], return_tensors="pt")
+with torch.no_grad():
+    probabilities = torch.sigmoid(model(**encoded).logits).numpy()[0]
+
+labels = [label for label, probability, threshold
+          in zip(config["labels"], probabilities, config["thresholds"])
+          if probability >= threshold]
+print(labels)   # ['frames_drop']
+```
+
+## Training
+
+- **Base model:** `distilbert-base-multilingual-cased`
+- **Objective:** `BCEWithLogitsLoss` with `pos_weight` to compensate for class imbalance
+- **Hyperparameters:** learning rate 8e-5 (grid over 2e-5 / 5e-5 / 8e-5), 5 epochs with best-epoch
+  selection by validation macro-F1, batch size 16, max sequence length 64
+- **Split:** iterative-stratified multi-label split 70 / 15 / 15 — 2760 / 584 / 612 examples
+- **Hardware:** Apple MPS, ~290 s per configuration
+
+## Evaluation
+
+Primary metric is **macro-F1** — it weights rare classes equally with frequent ones.
+
+| split | macro-F1 | micro-F1 | macro ROC-AUC | mAP |
+|---|---|---|---|---|
+| validation | 0.681 | 0.717 | 0.828 | 0.727 |
+| test | 0.620 | 0.678 | 0.817 | 0.685 |
+
+Per-class F1 on the test set: `ping_latency` 0.78, `unable_launch` 0.70,
+`mouse_keyboard_headset` 0.67, `frames_drop` 0.62, `game_bug` 0.49, `failed_save` 0.46.
+
+The model was selected on the validation set among seven approaches, from TF-IDF with classical
+classifiers to a zero-shot LLM. A most-frequent-class baseline scores 0.119 macro-F1.
+
+## Limitations
+
+- **Noisy labels.** Labels are self-reported by users in the cancellation form, so they are neither
+  consistent nor mutually exclusive. Part of what looks like model error is actually label noise.
+- **Rare classes.** `game_bug` and `failed_save` are recognised in fewer than half of real cases —
+  they are infrequent in the data and semantically overlap with the other classes.
+- **Very short texts.** Comments under 30 characters often carry too little signal.
+- **Single split, single seed.** Differences of 0.01–0.02 macro-F1 are within noise.
+
+## Training data
+
+Internal user feedback of a cloud-gaming service, collected at subscription cancellation
+(4 616 records, 3 956 after cleaning). The dataset is confidential and is **not published**.

config.json

@@ -0,0 +1,44 @@
+{
+  "activation": "gelu",
+  "architectures": [
+    "DistilBertForSequenceClassification"
+  ],
+  "attention_dropout": 0.1,
+  "bos_token_id": null,
+  "dim": 768,
+  "dropout": 0.1,
+  "dtype": "float32",
+  "eos_token_id": null,
+  "hidden_dim": 3072,
+  "id2label": {
+    "0": "LABEL_0",
+    "1": "LABEL_1",
+    "2": "LABEL_2",
+    "3": "LABEL_3",
+    "4": "LABEL_4",
+    "5": "LABEL_5"
+  },
+  "initializer_range": 0.02,
+  "label2id": {
+    "LABEL_0": 0,
+    "LABEL_1": 1,
+    "LABEL_2": 2,
+    "LABEL_3": 3,
+    "LABEL_4": 4,
+    "LABEL_5": 5
+  },
+  "max_position_embeddings": 512,
+  "model_type": "distilbert",
+  "n_heads": 12,
+  "n_layers": 6,
+  "output_past": true,
+  "pad_token_id": 0,
+  "problem_type": "multi_label_classification",
+  "qa_dropout": 0.1,
+  "seq_classif_dropout": 0.2,
+  "sinusoidal_pos_embds": false,
+  "tie_weights_": true,
+  "tie_word_embeddings": true,
+  "transformers_version": "5.14.1",
+  "vocab_size": 119547
+}

inference_config.json

@@ -0,0 +1,22 @@
+{
+  "labels": [
+    "ping_latency",
+    "frames_drop",
+    "unable_launch",
+    "mouse_keyboard_headset",
+    "game_bug",
+    "failed_save"
+  ],
+  "thresholds": [
+    0.45000000000000007,
+    0.75,
+    0.7000000000000001,
+    0.525,
+    0.55,
+    0.8
+  ],
+  "max_len": 64,
+  "base_checkpoint": "distilbert-base-multilingual-cased",
+  "learning_rate": 8e-05,
+  "val_macro_f1": 0.6812
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a69642be835b984194f25f976a26c8609a42984f590790d5562147d5f3137e82
+size 541329680

tokenizer_config.json

@@ -0,0 +1,15 @@
+{
+  "backend": "tokenizers",
+  "cls_token": "[CLS]",
+  "do_lower_case": false,
+  "is_local": false,
+  "local_files_only": false,
+  "mask_token": "[MASK]",
+  "model_max_length": 512,
+  "pad_token": "[PAD]",
+  "sep_token": "[SEP]",
+  "strip_accents": null,
+  "tokenize_chinese_chars": true,
+  "tokenizer_class": "BertTokenizer",
+  "unk_token": "[UNK]"
+}