README.md

---
license: apache-2.0
base_model: distilbert/distilbert-base-uncased
datasets:
- bitext/Bitext-customer-support-llm-chatbot-training-dataset
language:
- en
pipeline_tag: text-classification
tags:
- customer-support
- intent-classification
- distilbert
- text-classification
- ml-intern
metrics:
- accuracy
- f1
---

# DistilBERT Customer Support Ticket Classifier

Fine-tuned [distilbert/distilbert-base-uncased](https://huggingface.co/distilbert/distilbert-base-uncased) for classifying customer support tickets into **27 intent categories**.

## Model Description

This model classifies raw customer support ticket text into one of 27 issue-type intents, enabling automated routing, prioritisation, and analytics for customer support pipelines.

- **Architecture:** DistilBERT-base-uncased (66M parameters, 40% smaller and 60% faster than BERT-base while retaining 97% of its performance)
- **Task:** Multi-class text classification (27 classes)
- **Training data:** [Bitext Customer Support LLM Chatbot Training Dataset](https://huggingface.co/datasets/bitext/Bitext-customer-support-llm-chatbot-training-dataset) — 26,872 English utterances, nearly perfectly balanced across all classes (imbalance ratio: 1.05×)

## Supported Intent Classes

| ID | Intent | ID | Intent |
|---|---|---|---|
| 0 | cancel_order | 14 | edit_account |
| 1 | change_order | 15 | get_invoice |
| 2 | change_shipping_address | 16 | get_refund |
| 3 | check_cancellation_fee | 17 | newsletter_subscription |
| 4 | check_invoice | 18 | payment_issue |
| 5 | check_payment_methods | 19 | place_order |
| 6 | check_refund_policy | 20 | recover_password |
| 7 | complaint | 21 | registration_problems |
| 8 | contact_customer_service | 22 | review |
| 9 | contact_human_agent | 23 | set_up_shipping_address |
| 10 | create_account | 24 | switch_account |
| 11 | delete_account | 25 | track_order |
| 12 | delivery_options | 26 | track_refund |
| 13 | delivery_period | | |

## Training Configuration

| Hyperparameter | Value |
|---|---|
| Base model | distilbert/distilbert-base-uncased |
| Epochs | 3 |
| Batch size (per device) | 32 |
| Learning rate | 2e-5 |
| Weight decay | 0.01 |
| Warmup ratio | 0.1 |
| Max sequence length | 128 tokens |
| Best model selected by | Macro F1 |
| Optimizer | AdamW |
| Precision | fp16 |

## Evaluation Results

> ⏳ Model training pending. Accuracy and Macro F1 will be filled in after training completes.

Evaluated on a held-out test set (15% of data, ~4,031 samples):

| Metric | Value |
|---|---|
| Accuracy | *(pending)* |
| Macro F1 | *(pending)* |

See `confusion_matrix.png` for the full per-class breakdown (added after training).

## Usage

```python
from transformers import pipeline

classifier = pipeline(
    "text-classification",
    model="annebanne/distilbert-support-classifier",
)

# Single ticket
result = classifier("I need to cancel my order, it hasn't shipped yet.")
print(result)
# [{'label': 'cancel_order', 'score': 0.98}]

# Batch of tickets
tickets = [
    "Where is my refund? It's been 2 weeks.",
    "I can't log into my account after resetting my password.",
    "Please send me an invoice for order #12345.",
    "My payment keeps getting declined.",
]
results = classifier(tickets)
for ticket, res in zip(tickets, results):
    print(f"{res['label']:35s} ({res['score']:.2%}) — {ticket[:60]}")
```

## Dataset

- **Source:** [bitext/Bitext-customer-support-llm-chatbot-training-dataset](https://huggingface.co/datasets/bitext/Bitext-customer-support-llm-chatbot-training-dataset)
- **Size:** 26,872 utterances
- **Language:** English
- **Balance:** Near-perfect — each class has ~950–1,000 examples (imbalance ratio: 1.05×)
- **Split:** 85% train (22,841) / 15% test (4,031), random seed 42

## Limitations

- Trained on synthetic/augmented data. Real-world distribution may differ from production tickets.
- Performs best on short, single-intent utterances (similar to training data style).
- English only.
- Does not handle multi-intent tickets (predicts a single label).

## Citation

If you use this model, please cite the base model:

```bibtex
@article{sanh2019distilbert,
  title={DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter},
  author={Sanh, Victor and Debut, Lysandre and Chaumond, Julien and Wolf, Thomas},
  journal={arXiv preprint arXiv:1910.01108},
  year={2019}
}
```

<!-- ml-intern-provenance -->
## Generated by ML Intern

This model repository was generated by [ML Intern](https://github.com/huggingface/ml-intern), an agent for machine learning research and development on the Hugging Face Hub.

- Try ML Intern: https://smolagents-ml-intern.hf.space
- Source code: https://github.com/huggingface/ml-intern