| --- |
| license: mit |
| library_name: scikit-learn |
| tags: |
| - text-classification |
| - intent-classification |
| - customer-support |
| - banking |
| - banking77 |
| - customer-service |
| pipeline_tag: text-classification |
| --- |
| |
| # Customer Support AI |
|
|
| A machine learning model for classifying banking customer-support messages into their corresponding customer intent. |
|
|
| This project demonstrates a complete machine-learning workflow, including dataset inspection, data cleaning, train/validation/test splitting, model training, evaluation, error analysis, and prediction. |
|
|
| ## Project Overview |
|
|
| Customer-support systems receive a large number of messages every day. Automatically identifying the intent behind each message can help route customer queries to the correct support workflow. |
|
|
| This model takes a customer's message as input and predicts one of **77 banking-related customer-support intents**. |
|
|
| ### Example |
|
|
| **Input:** |
|
|
| ```text |
| My card hasn't arrived yet |
| ``` |
|
|
| **Prediction:** |
|
|
| ```text |
| card_arrival |
| ``` |
|
|
| Another example: |
|
|
| **Input:** |
|
|
| ```text |
| I want to cancel my transfer |
| ``` |
|
|
| **Prediction:** |
|
|
| ```text |
| cancel_transfer |
| ``` |
|
|
| --- |
|
|
| ## Task |
|
|
| **Text Classification / Customer Intent Classification** |
|
|
| The model performs multi-class classification on customer-support messages. |
|
|
| ### Input |
|
|
| A natural-language customer-support message. |
|
|
| ### Output |
|
|
| One of 77 predefined banking customer-support intents. |
|
|
| --- |
|
|
| ## Dataset |
|
|
| The model was trained using the **Banking77** dataset. |
|
|
| Banking77 contains banking-related customer queries categorized into **77 different intents**. |
|
|
| The dataset was processed through the following pipeline: |
|
|
| ```text |
| Raw Dataset |
| ↓ |
| Dataset Inspection |
| ↓ |
| Data Cleaning |
| ↓ |
| Train / Validation / Test Split |
| ↓ |
| Model Training |
| ↓ |
| Validation Evaluation |
| ↓ |
| Error Analysis |
| ↓ |
| Final Test Evaluation |
| ``` |
|
|
| --- |
|
|
| ## Model |
|
|
| The trained model is a **scikit-learn text-classification model** saved using `joblib`. |
|
|
| Model file: |
|
|
| ```text |
| customer_support_model.pkl |
| ``` |
|
|
| The model can be loaded in Python using: |
|
|
| ```python |
| import joblib |
| |
| model = joblib.load("customer_support_model.pkl") |
| |
| prediction = model.predict([ |
| "My card hasn't arrived yet" |
| ]) |
| |
| print(prediction) |
| ``` |
|
|
| Expected output: |
|
|
| ```text |
| ['card_arrival'] |
| ``` |
|
|
| --- |
|
|
| ## Model Performance |
|
|
| The model was evaluated using separate validation and test datasets. |
|
|
| ### Validation Results |
|
|
| Validation records: |
|
|
| ```text |
| 1,000 |
| ``` |
|
|
| Validation accuracy: |
|
|
| ```text |
| 83.90% |
| ``` |
|
|
| ### Final Test Results |
|
|
| Test records: |
|
|
| ```text |
| 1,000 |
| ``` |
|
|
| Test accuracy: |
|
|
| ```text |
| 84.70% |
| ``` |
|
|
| Additional test metrics: |
|
|
| | Metric | Score | |
| |---|---:| |
| | Accuracy | 84.70% | |
| | Macro F1 | 0.85 | |
| | Weighted F1 | 0.85 | |
|
|
| The final test evaluation was performed on data that was not used during model training. |
|
|
| --- |
|
|
| ## Error Analysis |
|
|
| Error analysis was performed on the validation dataset to understand where the model makes incorrect predictions. |
|
|
| ### Validation Results |
|
|
| ```text |
| Validation records: 1,000 |
| Correct predictions: 839 |
| Incorrect predictions: 161 |
| Error rate: 16.10% |
| ``` |
|
|
| The analysis showed that some errors occur between semantically similar customer-support intents. |
|
|
| For example, messages involving: |
|
|
| - pending payments |
| - failed payments |
| - reversed payments |
| - top-up problems |
| - card-related problems |
| - transfer-related problems |
|
|
| can sometimes contain similar language, making them harder to classify. |
|
|
| The project includes an error-analysis script that generates: |
|
|
| ```text |
| reports/validation_predictions.csv |
| ``` |
|
|
| --- |
|
|
| ## Supported Intent Categories |
|
|
| The model supports 77 banking customer-support intents, including: |
|
|
| ```text |
| Refund_not_showing_up |
| activate_my_card |
| age_limit |
| apple_pay_or_google_pay |
| atm_support |
| automatic_top_up |
| balance_not_updated_after_bank_transfer |
| balance_not_updated_after_cheque_or_cash_deposit |
| beneficiary_not_allowed |
| cancel_transfer |
| card_about_to_expire |
| card_acceptance |
| card_arrival |
| card_delivery_estimate |
| card_linking |
| card_not_working |
| card_payment_fee_charged |
| card_payment_not_recognised |
| card_payment_wrong_exchange_rate |
| card_swallowed |
| cash_withdrawal_charge |
| cash_withdrawal_not_recognised |
| change_pin |
| compromised_card |
| contactless_not_working |
| country_support |
| declined_card_payment |
| declined_cash_withdrawal |
| declined_transfer |
| direct_debit_payment_not_recognised |
| disposable_card_limits |
| edit_personal_details |
| exchange_charge |
| exchange_rate |
| exchange_via_app |
| extra_charge_on_statement |
| failed_transfer |
| fiat_currency_support |
| get_disposable_virtual_card |
| get_physical_card |
| getting_spare_card |
| getting_virtual_card |
| lost_or_stolen_card |
| lost_or_stolen_phone |
| order_physical_card |
| passcode_forgotten |
| pending_card_payment |
| pending_cash_withdrawal |
| pending_top_up |
| pending_transfer |
| pin_blocked |
| receiving_money |
| request_refund |
| reverted_card_payment? |
| supported_cards_and_currencies |
| terminate_account |
| top_up_by_bank_transfer_charge |
| top_up_by_card_charge |
| top_up_by_cash_or_cheque |
| top_up_failed |
| top_up_limits |
| top_up_reverted |
| topping_up_by_card |
| transaction_charged_twice |
| transfer_fee_charged |
| transfer_into_account |
| transfer_not_received_by_recipient |
| transfer_timing |
| unable_to_verify_identity |
| verify_my_identity |
| verify_source_of_funds |
| verify_top_up |
| virtual_card_not_working |
| visa_or_mastercard |
| why_verify_identity |
| wrong_amount_of_cash_received |
| wrong_exchange_rate_for_cash_withdrawal |
| ``` |
|
|
| --- |
|
|
| ## Project Features |
|
|
| The complete project contains: |
|
|
| - Dataset inspection |
| - Data cleaning |
| - Train/validation/test splitting |
| - Machine-learning model training |
| - Validation evaluation |
| - Classification report |
| - Error analysis |
| - Final test evaluation |
| - Interactive prediction |
| - Streamlit web application |
| - Model serialization using Joblib |
|
|
| --- |
|
|
| ## Project Structure |
|
|
| ```text |
| customer-support-annotation/ |
| │ |
| ├── annotation/ |
| │ |
| ├── data/ |
| │ ├── raw/ |
| │ │ └── train.csv |
| │ │ |
| │ └── processed/ |
| │ ├── cleaned_train.csv |
| │ ├── train.csv |
| │ ├── validation.csv |
| │ └── test.csv |
| │ |
| ├── models/ |
| │ └── customer_support_model.pkl |
| │ |
| ├── reports/ |
| │ └── validation_predictions.csv |
| │ |
| ├── scripts/ |
| │ ├── clean_dataset.py |
| │ ├── error_analysis.py |
| │ ├── evaluate_model.py |
| │ ├── inspect_dataset.py |
| │ ├── predict.py |
| │ ├── split_dataset.py |
| │ ├── test_model.py |
| │ └── train_model.py |
| │ |
| ├── app.py |
| ├── requirements.txt |
| └── README.md |
| ``` |
|
|
| --- |
|
|
| ## Interactive Prediction |
|
|
| The project includes an interactive prediction script. |
|
|
| Run: |
|
|
| ```bash |
| python scripts/predict.py |
| ``` |
|
|
| Example: |
|
|
| ```text |
| ===== CUSTOMER SUPPORT AI ===== |
| Type 'exit' to stop. |
| |
| Customer message: My card hasn't arrived yet |
| Predicted Intent: card_arrival |
| ``` |
|
|
| --- |
|
|
| ## Live Demo |
|
|
| The model is also integrated into a web application that allows users to enter customer-support messages and receive predicted intents. |
|
|
| **Live Demo:** |
|
|
| Add your Streamlit application URL here. |
|
|
| ```text |
| [https://customer-support-ai-gkmuxmcdqfjcwk3q6tpidm.streamlit.app/] |
| ``` |
|
|
| --- |
|
|
| ## Installation |
|
|
| Clone the project: |
|
|
| ```bash |
| git clone https://github.com/princechouhan3/customer-support-ai.git |
| cd customer-support-annotation |
| ``` |
|
|
| Install dependencies: |
|
|
| ```bash |
| pip install -r requirements.txt |
| ``` |
|
|
| Run the prediction application: |
|
|
| ```bash |
| python scripts/predict.py |
| ``` |
|
|
| --- |
|
|
| ## Source Code |
|
|
| The complete source code, training scripts, evaluation scripts, dataset-processing pipeline, and application code are available on GitHub. |
|
|
| **GitHub Repository:** |
|
|
| https://github.com/princechouhan3/customer-support-ai |
|
|
| --- |
|
|
| ## Potential Applications |
|
|
| This type of intent-classification model can be used as a component of: |
|
|
| - Banking customer-support systems |
| - Customer-service chatbots |
| - Automated ticket routing |
| - Support ticket classification |
| - Customer-support analytics |
| - FAQ and help-desk automation |
| - Intent detection systems |
|
|
| The model itself is a classification component and can be integrated into a larger customer-support or chatbot system. |
|
|
| --- |
|
|
| ## Limitations |
|
|
| This model was trained and evaluated using the Banking77 dataset. |
|
|
| Its performance on real-world customer messages may differ from the reported test performance. |
|
|
| The model should not be used as a production banking decision system without additional: |
|
|
| - domain-specific validation |
| - security testing |
| - monitoring |
| - bias evaluation |
| - robustness testing |
| - production testing |
| - human oversight where required |
|
|
| The model is intended primarily as a machine-learning project and portfolio demonstration. |
|
|
| --- |
|
|
| ## Future Improvements |
|
|
| Possible future improvements include: |
|
|
| - Improving classification accuracy |
| - Performing hyperparameter tuning |
| - Adding confidence scores |
| - Handling unknown or out-of-domain queries |
| - Improving error handling |
| - Adding more real-world customer-support data |
| - Comparing multiple ML algorithms |
| - Deploying the model as an API |
| - Integrating the classifier with a conversational AI system |
| - Adding monitoring and model evaluation pipelines |
|
|
| --- |
|
|
| ## Project |
|
|
| **Customer Support AI — Banking Intent Classification** |
|
|
| This project demonstrates an end-to-end machine-learning workflow from raw customer-support data to a trained and evaluated classification model and an interactive prediction application. |
|
|
| --- |
|
|
| ## License |
|
|
| This project is released under the MIT License. |
|
|
