Customer Support AI
A machine learning model for classifying banking customer-support messages into their corresponding customer intent.
This project demonstrates a complete machine-learning workflow, including dataset inspection, data cleaning, train/validation/test splitting, model training, evaluation, error analysis, and prediction.
Project Overview
Customer-support systems receive a large number of messages every day. Automatically identifying the intent behind each message can help route customer queries to the correct support workflow.
This model takes a customer's message as input and predicts one of 77 banking-related customer-support intents.
Example
Input:
My card hasn't arrived yet
Prediction:
card_arrival
Another example:
Input:
I want to cancel my transfer
Prediction:
cancel_transfer
Task
Text Classification / Customer Intent Classification
The model performs multi-class classification on customer-support messages.
Input
A natural-language customer-support message.
Output
One of 77 predefined banking customer-support intents.
Dataset
The model was trained using the Banking77 dataset.
Banking77 contains banking-related customer queries categorized into 77 different intents.
The dataset was processed through the following pipeline:
Raw Dataset
↓
Dataset Inspection
↓
Data Cleaning
↓
Train / Validation / Test Split
↓
Model Training
↓
Validation Evaluation
↓
Error Analysis
↓
Final Test Evaluation
Model
The trained model is a scikit-learn text-classification model saved using joblib.
Model file:
customer_support_model.pkl
The model can be loaded in Python using:
import joblib
model = joblib.load("customer_support_model.pkl")
prediction = model.predict([
"My card hasn't arrived yet"
])
print(prediction)
Expected output:
['card_arrival']
Model Performance
The model was evaluated using separate validation and test datasets.
Validation Results
Validation records:
1,000
Validation accuracy:
83.90%
Final Test Results
Test records:
1,000
Test accuracy:
84.70%
Additional test metrics:
| Metric | Score |
|---|---|
| Accuracy | 84.70% |
| Macro F1 | 0.85 |
| Weighted F1 | 0.85 |
The final test evaluation was performed on data that was not used during model training.
Error Analysis
Error analysis was performed on the validation dataset to understand where the model makes incorrect predictions.
Validation Results
Validation records: 1,000
Correct predictions: 839
Incorrect predictions: 161
Error rate: 16.10%
The analysis showed that some errors occur between semantically similar customer-support intents.
For example, messages involving:
- pending payments
- failed payments
- reversed payments
- top-up problems
- card-related problems
- transfer-related problems
can sometimes contain similar language, making them harder to classify.
The project includes an error-analysis script that generates:
reports/validation_predictions.csv
Supported Intent Categories
The model supports 77 banking customer-support intents, including:
Refund_not_showing_up
activate_my_card
age_limit
apple_pay_or_google_pay
atm_support
automatic_top_up
balance_not_updated_after_bank_transfer
balance_not_updated_after_cheque_or_cash_deposit
beneficiary_not_allowed
cancel_transfer
card_about_to_expire
card_acceptance
card_arrival
card_delivery_estimate
card_linking
card_not_working
card_payment_fee_charged
card_payment_not_recognised
card_payment_wrong_exchange_rate
card_swallowed
cash_withdrawal_charge
cash_withdrawal_not_recognised
change_pin
compromised_card
contactless_not_working
country_support
declined_card_payment
declined_cash_withdrawal
declined_transfer
direct_debit_payment_not_recognised
disposable_card_limits
edit_personal_details
exchange_charge
exchange_rate
exchange_via_app
extra_charge_on_statement
failed_transfer
fiat_currency_support
get_disposable_virtual_card
get_physical_card
getting_spare_card
getting_virtual_card
lost_or_stolen_card
lost_or_stolen_phone
order_physical_card
passcode_forgotten
pending_card_payment
pending_cash_withdrawal
pending_top_up
pending_transfer
pin_blocked
receiving_money
request_refund
reverted_card_payment?
supported_cards_and_currencies
terminate_account
top_up_by_bank_transfer_charge
top_up_by_card_charge
top_up_by_cash_or_cheque
top_up_failed
top_up_limits
top_up_reverted
topping_up_by_card
transaction_charged_twice
transfer_fee_charged
transfer_into_account
transfer_not_received_by_recipient
transfer_timing
unable_to_verify_identity
verify_my_identity
verify_source_of_funds
verify_top_up
virtual_card_not_working
visa_or_mastercard
why_verify_identity
wrong_amount_of_cash_received
wrong_exchange_rate_for_cash_withdrawal
Project Features
The complete project contains:
- Dataset inspection
- Data cleaning
- Train/validation/test splitting
- Machine-learning model training
- Validation evaluation
- Classification report
- Error analysis
- Final test evaluation
- Interactive prediction
- Streamlit web application
- Model serialization using Joblib
Project Structure
customer-support-annotation/
│
├── annotation/
│
├── data/
│ ├── raw/
│ │ └── train.csv
│ │
│ └── processed/
│ ├── cleaned_train.csv
│ ├── train.csv
│ ├── validation.csv
│ └── test.csv
│
├── models/
│ └── customer_support_model.pkl
│
├── reports/
│ └── validation_predictions.csv
│
├── scripts/
│ ├── clean_dataset.py
│ ├── error_analysis.py
│ ├── evaluate_model.py
│ ├── inspect_dataset.py
│ ├── predict.py
│ ├── split_dataset.py
│ ├── test_model.py
│ └── train_model.py
│
├── app.py
├── requirements.txt
└── README.md
Interactive Prediction
The project includes an interactive prediction script.
Run:
python scripts/predict.py
Example:
===== CUSTOMER SUPPORT AI =====
Type 'exit' to stop.
Customer message: My card hasn't arrived yet
Predicted Intent: card_arrival
Live Demo
The model is also integrated into a web application that allows users to enter customer-support messages and receive predicted intents.
Live Demo:
Add your Streamlit application URL here.
[https://customer-support-ai-gkmuxmcdqfjcwk3q6tpidm.streamlit.app/]
Installation
Clone the project:
git clone https://github.com/princechouhan3/customer-support-ai.git
cd customer-support-annotation
Install dependencies:
pip install -r requirements.txt
Run the prediction application:
python scripts/predict.py
Source Code
The complete source code, training scripts, evaluation scripts, dataset-processing pipeline, and application code are available on GitHub.
GitHub Repository:
https://github.com/princechouhan3/customer-support-ai
Potential Applications
This type of intent-classification model can be used as a component of:
- Banking customer-support systems
- Customer-service chatbots
- Automated ticket routing
- Support ticket classification
- Customer-support analytics
- FAQ and help-desk automation
- Intent detection systems
The model itself is a classification component and can be integrated into a larger customer-support or chatbot system.
Limitations
This model was trained and evaluated using the Banking77 dataset.
Its performance on real-world customer messages may differ from the reported test performance.
The model should not be used as a production banking decision system without additional:
- domain-specific validation
- security testing
- monitoring
- bias evaluation
- robustness testing
- production testing
- human oversight where required
The model is intended primarily as a machine-learning project and portfolio demonstration.
Future Improvements
Possible future improvements include:
- Improving classification accuracy
- Performing hyperparameter tuning
- Adding confidence scores
- Handling unknown or out-of-domain queries
- Improving error handling
- Adding more real-world customer-support data
- Comparing multiple ML algorithms
- Deploying the model as an API
- Integrating the classifier with a conversational AI system
- Adding monitoring and model evaluation pipelines
Project
Customer Support AI — Banking Intent Classification
This project demonstrates an end-to-end machine-learning workflow from raw customer-support data to a trained and evaluated classification model and an interactive prediction application.
License
This project is released under the MIT License.