--- license: mit library_name: scikit-learn tags: - text-classification - intent-classification - customer-support - banking - banking77 - customer-service pipeline_tag: text-classification --- # Customer Support AI A machine learning model for classifying banking customer-support messages into their corresponding customer intent. This project demonstrates a complete machine-learning workflow, including dataset inspection, data cleaning, train/validation/test splitting, model training, evaluation, error analysis, and prediction. ## Project Overview Customer-support systems receive a large number of messages every day. Automatically identifying the intent behind each message can help route customer queries to the correct support workflow. This model takes a customer's message as input and predicts one of **77 banking-related customer-support intents**. ### Example **Input:** ```text My card hasn't arrived yet ``` **Prediction:** ```text card_arrival ``` Another example: **Input:** ```text I want to cancel my transfer ``` **Prediction:** ```text cancel_transfer ``` --- ## Task **Text Classification / Customer Intent Classification** The model performs multi-class classification on customer-support messages. ### Input A natural-language customer-support message. ### Output One of 77 predefined banking customer-support intents. --- ## Dataset The model was trained using the **Banking77** dataset. Banking77 contains banking-related customer queries categorized into **77 different intents**. The dataset was processed through the following pipeline: ```text Raw Dataset ↓ Dataset Inspection ↓ Data Cleaning ↓ Train / Validation / Test Split ↓ Model Training ↓ Validation Evaluation ↓ Error Analysis ↓ Final Test Evaluation ``` --- ## Model The trained model is a **scikit-learn text-classification model** saved using `joblib`. Model file: ```text customer_support_model.pkl ``` The model can be loaded in Python using: ```python import joblib model = joblib.load("customer_support_model.pkl") prediction = model.predict([ "My card hasn't arrived yet" ]) print(prediction) ``` Expected output: ```text ['card_arrival'] ``` --- ## Model Performance The model was evaluated using separate validation and test datasets. ### Validation Results Validation records: ```text 1,000 ``` Validation accuracy: ```text 83.90% ``` ### Final Test Results Test records: ```text 1,000 ``` Test accuracy: ```text 84.70% ``` Additional test metrics: | Metric | Score | |---|---:| | Accuracy | 84.70% | | Macro F1 | 0.85 | | Weighted F1 | 0.85 | The final test evaluation was performed on data that was not used during model training. --- ## Error Analysis Error analysis was performed on the validation dataset to understand where the model makes incorrect predictions. ### Validation Results ```text Validation records: 1,000 Correct predictions: 839 Incorrect predictions: 161 Error rate: 16.10% ``` The analysis showed that some errors occur between semantically similar customer-support intents. For example, messages involving: - pending payments - failed payments - reversed payments - top-up problems - card-related problems - transfer-related problems can sometimes contain similar language, making them harder to classify. The project includes an error-analysis script that generates: ```text reports/validation_predictions.csv ``` --- ## Supported Intent Categories The model supports 77 banking customer-support intents, including: ```text Refund_not_showing_up activate_my_card age_limit apple_pay_or_google_pay atm_support automatic_top_up balance_not_updated_after_bank_transfer balance_not_updated_after_cheque_or_cash_deposit beneficiary_not_allowed cancel_transfer card_about_to_expire card_acceptance card_arrival card_delivery_estimate card_linking card_not_working card_payment_fee_charged card_payment_not_recognised card_payment_wrong_exchange_rate card_swallowed cash_withdrawal_charge cash_withdrawal_not_recognised change_pin compromised_card contactless_not_working country_support declined_card_payment declined_cash_withdrawal declined_transfer direct_debit_payment_not_recognised disposable_card_limits edit_personal_details exchange_charge exchange_rate exchange_via_app extra_charge_on_statement failed_transfer fiat_currency_support get_disposable_virtual_card get_physical_card getting_spare_card getting_virtual_card lost_or_stolen_card lost_or_stolen_phone order_physical_card passcode_forgotten pending_card_payment pending_cash_withdrawal pending_top_up pending_transfer pin_blocked receiving_money request_refund reverted_card_payment? supported_cards_and_currencies terminate_account top_up_by_bank_transfer_charge top_up_by_card_charge top_up_by_cash_or_cheque top_up_failed top_up_limits top_up_reverted topping_up_by_card transaction_charged_twice transfer_fee_charged transfer_into_account transfer_not_received_by_recipient transfer_timing unable_to_verify_identity verify_my_identity verify_source_of_funds verify_top_up virtual_card_not_working visa_or_mastercard why_verify_identity wrong_amount_of_cash_received wrong_exchange_rate_for_cash_withdrawal ``` --- ## Project Features The complete project contains: - Dataset inspection - Data cleaning - Train/validation/test splitting - Machine-learning model training - Validation evaluation - Classification report - Error analysis - Final test evaluation - Interactive prediction - Streamlit web application - Model serialization using Joblib --- ## Project Structure ```text customer-support-annotation/ │ ├── annotation/ │ ├── data/ │ ├── raw/ │ │ └── train.csv │ │ │ └── processed/ │ ├── cleaned_train.csv │ ├── train.csv │ ├── validation.csv │ └── test.csv │ ├── models/ │ └── customer_support_model.pkl │ ├── reports/ │ └── validation_predictions.csv │ ├── scripts/ │ ├── clean_dataset.py │ ├── error_analysis.py │ ├── evaluate_model.py │ ├── inspect_dataset.py │ ├── predict.py │ ├── split_dataset.py │ ├── test_model.py │ └── train_model.py │ ├── app.py ├── requirements.txt └── README.md ``` --- ## Interactive Prediction The project includes an interactive prediction script. Run: ```bash python scripts/predict.py ``` Example: ```text ===== CUSTOMER SUPPORT AI ===== Type 'exit' to stop. Customer message: My card hasn't arrived yet Predicted Intent: card_arrival ``` --- ## Live Demo The model is also integrated into a web application that allows users to enter customer-support messages and receive predicted intents. **Live Demo:** Add your Streamlit application URL here. ```text [https://customer-support-ai-gkmuxmcdqfjcwk3q6tpidm.streamlit.app/] ``` --- ## Installation Clone the project: ```bash git clone https://github.com/princechouhan3/customer-support-ai.git cd customer-support-annotation ``` Install dependencies: ```bash pip install -r requirements.txt ``` Run the prediction application: ```bash python scripts/predict.py ``` --- ## Source Code The complete source code, training scripts, evaluation scripts, dataset-processing pipeline, and application code are available on GitHub. **GitHub Repository:** https://github.com/princechouhan3/customer-support-ai --- ## Potential Applications This type of intent-classification model can be used as a component of: - Banking customer-support systems - Customer-service chatbots - Automated ticket routing - Support ticket classification - Customer-support analytics - FAQ and help-desk automation - Intent detection systems The model itself is a classification component and can be integrated into a larger customer-support or chatbot system. --- ## Limitations This model was trained and evaluated using the Banking77 dataset. Its performance on real-world customer messages may differ from the reported test performance. The model should not be used as a production banking decision system without additional: - domain-specific validation - security testing - monitoring - bias evaluation - robustness testing - production testing - human oversight where required The model is intended primarily as a machine-learning project and portfolio demonstration. --- ## Future Improvements Possible future improvements include: - Improving classification accuracy - Performing hyperparameter tuning - Adding confidence scores - Handling unknown or out-of-domain queries - Improving error handling - Adding more real-world customer-support data - Comparing multiple ML algorithms - Deploying the model as an API - Integrating the classifier with a conversational AI system - Adding monitoring and model evaluation pipelines --- ## Project **Customer Support AI — Banking Intent Classification** This project demonstrates an end-to-end machine-learning workflow from raw customer-support data to a trained and evaluated classification model and an interactive prediction application. --- ## License This project is released under the MIT License.