Text Classification
Transformers
English
distilbert
it-support
ticket-routing
text-embeddings-inference

DistilBERT for IT Support Question Routing (5 classes)

distilbert-base-uncased fine-tuned to route real IT-support questions into five queues. Reaches 0.893 accuracy / 0.893 macro-F1, roughly on par with a TF-IDF + Logistic Regression baseline.

Model Details

  • Author: Vishwajeet Adkine
  • Base model: distilbert-base-uncased (Apache-2.0)
  • Task: single-label text classification
  • Classes: Linux, Apple, Android, Server & Network, Security
  • Input: question title plus body, truncated to 256 tokens

Intended Use

Prototype for routing support questions to the right team, and a worked example of fine-tuning a transformer on real (non-synthetic) text with an honest baseline.

Not intended for production ticket routing without evaluation on your own tickets, or for any automated decision about people.

Training Data

Real, human-written questions from five Stack Exchange communities (Ask Ubuntu, Ask Different, Android Enthusiasts, Server Fault, Information Security), taken from the flax-sentence-embeddings/stackexchange_titlebody_best_voted_answer_jsonl dataset. Up to 4,000 questions per class, cleaned and de-duplicated, split 80 / 10 / 10 (stratified).

Important: the label is the community a question was posted in, not a hand-annotated ticket category. Real labelled IT tickets are rarely public, so this is a proxy.

Training Procedure

Hugging Face Trainer: 3 epochs, batch size 32, learning rate 2e-5, 10% warm-up, weight decay 0.01, max length 256, fp16, best epoch by validation macro-F1.

Results (held-out test set)

Model Accuracy Macro-F1
TF-IDF + Logistic Regression 0.8940 0.8941
TF-IDF, brand words masked 0.8605 0.8604
DistilBERT (fine-tuned) 0.8930 0.8927

What this shows: on this task the fine-tuned transformer does not beat a strong bag-of-words baseline, so its extra cost is not justified by accuracy alone. Masking brand words ("Ubuntu", "iPhone", "Android"...) costs the baseline about 3.4 points, so part of the signal is obvious keywords. The masked-text experiment was not run on DistilBERT.

Limitations and Bias

  • Labels are communities, so some questions fit several classes (for example a VPN problem on macOS), which caps achievable accuracy.
  • Stack Exchange writing style differs from internal tickets (shorter, informal, multilingual, templated). Expect a drop on real company tickets.
  • Covers five classes only; no Windows, cloud, database or hardware categories.
  • English only. Reflects the demographics and topics of Stack Exchange users.
  • Low-confidence predictions should go to human triage.

How to Use

from transformers import pipeline
clf = pipeline("text-classification", model="<path-or-hub-id>", top_k=None)
clf("Nginx returns 502 after moving the backend to a new subnet.")

Licence

Base model: Apache-2.0. Training text comes from Stack Exchange (CC BY-SA), and the dataset card lists a non-commercial licence, so check terms and keep attribution before any commercial use. The weights are therefore marked other.

Downloads last month
16
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for vishwajeet456/DistilBERTforITSupport

Finetuned
(12,884)
this model

Dataset used to train vishwajeet456/DistilBERTforITSupport