|
|
--- |
|
|
library_name: transformers |
|
|
tags: [] |
|
|
--- |
|
|
|
|
|
# MagicSupport Intent Classifier (BERT Fine-Tuned) |
|
|
|
|
|
## Overview |
|
|
|
|
|
This model is a fine-tuned `bert-base-uncased` model for multi-class intent classification in customer support environments. |
|
|
|
|
|
It is optimized for: |
|
|
|
|
|
* Fast inference |
|
|
* High accuracy |
|
|
* Low deployment cost |
|
|
* Production-ready intent routing for support systems |
|
|
|
|
|
The model is designed for the MagicSupport platform but is generalizable to structured customer support intent detection tasks. |
|
|
|
|
|
--- |
|
|
|
|
|
## Model Details |
|
|
|
|
|
* Base Model: `bert-base-uncased` |
|
|
* Architecture: `BertForSequenceClassification` |
|
|
* Task: Multi-class intent classification |
|
|
* Number of Intents: 28 |
|
|
* Training Dataset: `bitext/Bitext-customer-support-llm-chatbot-training-dataset` |
|
|
* Loss: CrossEntropy with class weights |
|
|
* Framework: Hugging Face Transformers (PyTorch) |
|
|
|
|
|
--- |
|
|
|
|
|
## Performance |
|
|
|
|
|
### Validation Metrics (Epoch 5) |
|
|
|
|
|
* Accuracy: **0.9983** |
|
|
* F1 Micro: **0.9983** |
|
|
* F1 Macro: **0.9983** |
|
|
* Validation Loss: **0.0087** |
|
|
|
|
|
The model demonstrates strong generalization and stable convergence across 5 epochs. |
|
|
|
|
|
--- |
|
|
|
|
|
## Example Predictions |
|
|
|
|
|
| Query | Predicted Intent | Confidence | |
|
|
| ------------------------------------- | ---------------- | ---------- | |
|
|
| I want to cancel my order | cancel_order | 0.999 | |
|
|
| How do I track my shipment | delivery_options | 0.997 | |
|
|
| I need a refund for my purchase | get_refund | 0.999 | |
|
|
| I forgot my password | recover_password | 0.999 | |
|
|
| I have a complaint about your service | complaint | 0.996 | |
|
|
| hello | FALLBACK | 0.999 | |
|
|
|
|
|
The model also correctly identifies low-information inputs and maps them to a fallback intent. |
|
|
|
|
|
--- |
|
|
|
|
|
## Intended Use |
|
|
|
|
|
This model is intended for: |
|
|
|
|
|
* Customer support intent classification |
|
|
* Chatbot routing |
|
|
* Support ticket categorization |
|
|
* Voice-to-intent pipelines (after STT) |
|
|
* Pre-routing before LLM or RAG systems |
|
|
|
|
|
Typical production flow: |
|
|
|
|
|
User Query → BERT Intent Classifier → Route to: |
|
|
|
|
|
* Knowledge Base Retrieval |
|
|
* Ticketing System |
|
|
* Escalation to Human |
|
|
* Fallback LLM |
|
|
|
|
|
--- |
|
|
|
|
|
## Example Usage |
|
|
|
|
|
```python |
|
|
from transformers import AutoTokenizer, AutoModelForSequenceClassification |
|
|
import torch |
|
|
|
|
|
# Load model and tokenizer from HuggingFace Hub |
|
|
model_name = "learn-abc/magicSupport-intent-classifier" |
|
|
tokenizer = AutoTokenizer.from_pretrained(model_name) |
|
|
model = AutoModelForSequenceClassification.from_pretrained(model_name) |
|
|
|
|
|
# Set device |
|
|
device = torch.device("cuda" if torch.cuda.is_available() else "cpu") |
|
|
model.to(device) |
|
|
model.eval() |
|
|
|
|
|
# Prediction function |
|
|
def predict_intent(text, confidence_threshold=0.75): |
|
|
inputs = tokenizer(text, return_tensors="pt", truncation=True, max_length=64) |
|
|
inputs = {k: v.to(device) for k, v in inputs.items()} |
|
|
|
|
|
with torch.no_grad(): |
|
|
outputs = model(**inputs) |
|
|
logits = outputs.logits |
|
|
probs = torch.softmax(logits, dim=-1) |
|
|
confidence, prediction = torch.max(probs, dim=-1) |
|
|
|
|
|
predicted_intent = model.config.id2label[prediction.item()] |
|
|
confidence_score = confidence.item() |
|
|
|
|
|
# Apply confidence threshold |
|
|
if confidence_score < confidence_threshold: |
|
|
predicted_intent = "FALLBACK" |
|
|
|
|
|
return { |
|
|
"intent": predicted_intent, |
|
|
"confidence": confidence_score |
|
|
} |
|
|
|
|
|
# Example usage |
|
|
queries = [ |
|
|
"I want to cancel my order", |
|
|
"How do I track my package", |
|
|
"I need a refund", |
|
|
"hello there" |
|
|
] |
|
|
|
|
|
for query in queries: |
|
|
result = predict_intent(query) |
|
|
print(f"Query: {query}") |
|
|
print(f"Intent: {result['intent']}") |
|
|
print(f"Confidence: {result['confidence']:.3f}\n") |
|
|
``` |
|
|
|
|
|
--- |
|
|
|
|
|
## Design Decisions |
|
|
|
|
|
* BERT selected over larger LLMs for: |
|
|
|
|
|
* Low latency |
|
|
* Cost efficiency |
|
|
* Predictable inference |
|
|
* Edge deployability |
|
|
|
|
|
* Class weighting applied to mitigate dataset imbalance. |
|
|
|
|
|
* High confidence outputs indicate strong separation between intent classes. |
|
|
|
|
|
--- |
|
|
|
|
|
## Known Limitations |
|
|
|
|
|
* Designed for structured customer support queries. |
|
|
* May struggle with: |
|
|
|
|
|
* Highly conversational multi-turn context |
|
|
* Extremely domain-specific enterprise terminology |
|
|
* Heavy slang or multilingual input |
|
|
* Not trained for open-domain conversation. |
|
|
|
|
|
--- |
|
|
|
|
|
## Future Improvements |
|
|
|
|
|
* Add MagicSupport real production data for domain adaptation. |
|
|
* Add hierarchical intent structure. |
|
|
* Introduce confidence threshold calibration. |
|
|
* Add OOD (Out-of-Distribution) detection. |
|
|
* Quantized inference version for edge deployment. |
|
|
|
|
|
--- |
|
|
|
|
|
## License |
|
|
|
|
|
Specify your intended license here (e.g., MIT, Apache-2.0). |
|
|
|
|
|
--- |
|
|
|
|
|
## Citation |
|
|
|
|
|
If using this model in research or production, please cite appropriately. |
|
|
|
|
|
--- |
|
|
|
|
|
## Model Card Author |
|
|
|
|
|
For any inquiries or support, please reach out to: |
|
|
|
|
|
* **Author:** [Abhishek Singh](https://github.com/SinghIsWriting/) |
|
|
* **LinkedIn:** [My LinkedIn Profile](https://www.linkedin.com/in/abhishek-singh-bba2662a9) |
|
|
* **Portfolio:** [Abhishek Singh Portfolio](https://portfolio-abhishek-singh-nine.vercel.app/) |
|
|
|
