Add dual dataset support and update system documentation

- Enhanced train.py to support both VNTC and UTS2017_Bank datasets
- Added --dataset parameter for dataset selection
- Updated System Card with UTS2017_Bank performance metrics
- Achieved 92.33% accuracy on VNTC and 70.96% on UTS2017_Bank
- Cleaned up redundant files and old training runs

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>

Sonar Core 1 - System Card.md

@@ -6,17 +6,22 @@
 
 # Changelog
 
+**2025-09-27**
+
+- Added support for UTS2017_Bank Vietnamese banking text classification dataset
+- Achieved 70.96% accuracy on 14 banking service categories
+
 **2025-09-21**
 
 - Initial release of Sonar Core 1
 
 # Abstract
 
-**Sonar Core 1** is a machine learning-based text classification model designed for Vietnamese language processing. Built on a **TF-IDF** (Term Frequency-Inverse Document Frequency) feature extraction pipeline combined with **Logistic Regression**, this model achieves **92.33% accuracy** on the VNTC (Vietnamese Text Classification) dataset across **10 news categories**. The model is specifically designed for Vietnamese news article classification, content categorization for Vietnamese text, and document organization and tagging. Developed as a base model to provide quick and reliable text classification support for **scikit-learn >=1.6** integration since **underthesea 8.1.0**, it employs optimized feature engineering with **20,000 max features** and bigram support, along with a hash-based caching system for efficient processing. This system card provides comprehensive documentation of the model's architecture, performance metrics, intended uses, and limitations.
+**Sonar Core 1** is a machine learning-based text classification model designed for Vietnamese language processing. Built on a **TF-IDF** (Term Frequency-Inverse Document Frequency) feature extraction pipeline combined with **Logistic Regression**, this model achieves **92.33% accuracy** on the VNTC (Vietnamese Text Classification) dataset across **10 news categories** and **70.96% accuracy** on the UTS2017_Bank dataset across **14 banking service categories**. The model is specifically designed for Vietnamese news article classification, banking text categorization, content categorization for Vietnamese text, and document organization and tagging. Developed as a base model to provide quick and reliable text classification support for **scikit-learn >=1.6** integration since **underthesea 8.1.0**, it employs optimized feature engineering with **20,000 max features** and bigram support, along with a hash-based caching system for efficient processing. This system card provides comprehensive documentation of the model's architecture, performance metrics, intended uses, and limitations.
 
 # 1. Model Details
 
-**Sonar Core 1** is a Vietnamese text classification model built on **scikit-learn >=1.6**, utilizing a TF-IDF pipeline with Logistic Regression to classify text across 10 news categories. The architecture employs:
+**Sonar Core 1** is a Vietnamese text classification model built on **scikit-learn >=1.6**, utilizing a TF-IDF pipeline with Logistic Regression to classify text across multiple domains including news categories and banking services. The architecture employs:
 - CountVectorizer with **20,000 max features** (optimized from the initial 10,000)
 - N-gram extraction: unigram and bigram support
 - TF-IDF transformation with IDF weighting
@@ -27,7 +32,7 @@ Released on **2025-09-21**, the model achieves **92.33% test accuracy** and **95
 
 # 2. Training Data
 
-## 2.1 Supported Categories (10 classes)
+## 2.1 VNTC Dataset - News Categories (10 classes)
 1. **chinh_tri_xa_hoi** - Politics and Society
 2. **doi_song** - Lifestyle
 3. **khoa_hoc** - Science
@@ -39,27 +44,51 @@ Released on **2025-09-21**, the model achieves **92.33% test accuracy** and **95
 9. **van_hoa** - Culture
 10. **vi_tinh** - Information Technology
 
-## 2.2 Dataset
+## 2.2 UTS2017_Bank Dataset - Banking Categories (14 classes)
+1. **ACCOUNT** - Account services
+2. **CARD** - Card services
+3. **CUSTOMER_SUPPORT** - Customer support
+4. **DISCOUNT** - Discount offers
+5. **INTEREST_RATE** - Interest rate information
+6. **INTERNET_BANKING** - Internet banking services
+7. **LOAN** - Loan services
+8. **MONEY_TRANSFER** - Money transfer services
+9. **OTHER** - Other services
+10. **PAYMENT** - Payment services
+11. **PROMOTION** - Promotional offers
+12. **SAVING** - Savings accounts
+13. **SECURITY** - Security features
+14. **TRADEMARK** - Trademark/branding
+
+## 2.3 Dataset Details
+
+### VNTC Dataset
 - **Name**: VNTC (Vietnamese Text Classification) Dataset
 - **Training Samples**: 33,759 documents
 - **Test Samples**: 50,373 documents
 - **Language**: Vietnamese
 - **Format**: FastText format (__label__category followed by text)
+- **Distribution**: Balanced across 10 news categories
+- **Average document length**: ~200-500 words
 
-## 2.3 Data Distribution
-- Balanced across 10 news categories
-- Text preprocessing: None (raw Vietnamese text)
-- Average document length: ~200-500 words
+### UTS2017_Bank Dataset
+- **Name**: UTS2017_Bank Classification Dataset
+- **Training Samples**: 1,581 documents
+- **Test Samples**: 396 documents
+- **Language**: Vietnamese
+- **Format**: Text with categorical labels
+- **Distribution**: Imbalanced (CUSTOMER_SUPPORT: 39%, TRADEMARK: 35%, others: 26%)
+- **Text preprocessing**: None (raw Vietnamese text)
 
 # 3. Performance Metrics
 
-## 3.1 Overall Performance (Latest Results - 2025-09-21)
+## 3.1 VNTC Dataset Performance (2025-09-21)
 - **Training Accuracy**: 95.39%
 - **Test Accuracy**: 92.33%
 - **Training Time**: ~27.18 seconds (with caching system)
 - **Inference Time**: ~19.34 seconds for 50,373 samples
 
-## 3.2 Per-Class Performance (Latest Run - All 10 Classes)
+## 3.2 Per-Class Performance - VNTC Dataset
 | Category | Precision | Recall | F1-Score | Support |
 |----------|-----------|---------|-----------|---------|
 | chinh_tri_xa_hoi | 0.86 | 0.93 | 0.89 | 7,567 |
@@ -73,34 +102,82 @@ Released on **2025-09-21**, the model achieves **92.33% test accuracy** and **95
 | van_hoa | 0.93 | 0.95 | 0.94 | 6,250 |
 | vi_tinh | 0.94 | 0.95 | 0.94 | 4,560 |
 
-## 3.3 Aggregate Metrics (Latest Run)
+## 3.3 UTS2017_Bank Dataset Performance (2025-09-27)
+- **Training Accuracy**: 76.22%
+- **Test Accuracy**: 70.96%
+- **Training Time**: ~0.78 seconds
+- **Inference Time**: ~0.01 seconds for 396 samples
+
+## 3.4 Per-Class Performance - UTS2017_Bank Dataset
+| Category | Precision | Recall | F1-Score | Support |
+|----------|-----------|---------|-----------|---------|
+| ACCOUNT | 0.00 | 0.00 | 0.00 | 1 |
+| CARD | 0.00 | 0.00 | 0.00 | 13 |
+| CUSTOMER_SUPPORT | 0.62 | 0.97 | 0.76 | 155 |
+| DISCOUNT | 0.00 | 0.00 | 0.00 | 8 |
+| INTEREST_RATE | 0.50 | 0.08 | 0.14 | 12 |
+| INTERNET_BANKING | 0.00 | 0.00 | 0.00 | 14 |
+| LOAN | 0.67 | 0.13 | 0.22 | 15 |
+| MONEY_TRANSFER | 0.00 | 0.00 | 0.00 | 7 |
+| OTHER | 0.50 | 0.07 | 0.12 | 14 |
+| PAYMENT | 0.00 | 0.00 | 0.00 | 3 |
+| PROMOTION | 1.00 | 0.18 | 0.31 | 11 |
+| SAVING | 0.00 | 0.00 | 0.00 | 2 |
+| SECURITY | 0.00 | 0.00 | 0.00 | 1 |
+| TRADEMARK | 0.87 | 0.89 | 0.88 | 140 |
+
+## 3.5 Aggregate Metrics
+
+### VNTC Dataset
 - **Overall Accuracy**: 92%
 - **Macro Average**: Precision: 0.91, Recall: 0.90, F1: 0.91
 - **Weighted Average**: Precision: 0.92, Recall: 0.92, F1: 0.92
 
-## 3.4 Performance Analysis
+### UTS2017_Bank Dataset
+- **Overall Accuracy**: 71%
+- **Macro Average**: Precision: 0.30, Recall: 0.17, F1: 0.17
+- **Weighted Average**: Precision: 0.64, Recall: 0.71, F1: 0.63
+
+## 3.6 Performance Analysis
+
+### VNTC Dataset
 - **Best Performing Categories**: Sports (the_thao) achieves 98% F1-score, followed by Health, World, Culture, and IT (all 94% F1-score)
 - **Lowest Performing Category**: Lifestyle (doi_song) with 76% F1-score due to lower recall (71%)
+
+### UTS2017_Bank Dataset
+- **Best Performing Categories**: TRADEMARK (88% F1-score) and CUSTOMER_SUPPORT (76% F1-score)
+- **Challenges**: Many minority classes with insufficient training data result in zero predictions
+- **Data Imbalance**: Significant class imbalance with CUSTOMER_SUPPORT and TRADEMARK dominating (74% of data)
+
+### General Observations
 - **Feature Count**: Uses 20,000 max features with bigram support
 - **Caching System**: Hash-based caching for efficient vectorizer and TF-IDF processing
+- **Model performs better on balanced datasets** (VNTC) compared to imbalanced ones (UTS2017_Bank)
 
 # 4. Limitations
 
 ## 4.1 Known Limitations
 1. **Language Specificity**: Only works with Vietnamese text
-2. **Domain Specificity**: Optimized for news articles, may not perform well on:
-   - Social media posts
-   - Technical documentation
+2. **Domain Specificity**: Optimized for specific domains, may not generalize well to:
+   - Social media posts (unless trained on specific datasets)
+   - Technical documentation outside IT/banking domains
    - Conversational text
 3. **Feature Limitations**:
    - Limited to 20,000 most frequent features
    - May miss rare but important terms
-4. **Class Confusion**: Lower performance on lifestyle (doi_song) category (71% recall)
+4. **Class Imbalance Sensitivity**:
+   - Performance degrades significantly with imbalanced datasets
+   - Minority classes may receive zero predictions (as seen in UTS2017_Bank)
+5. **Specific Category Weaknesses**:
+   - VNTC: Lower performance on lifestyle (doi_song) category (71% recall)
+   - UTS2017_Bank: Poor performance on minority classes (ACCOUNT, CARD, PAYMENT, etc.)
 
 ## 4.2 Biases
-- Trained on news articles which may have formal writing style bias
-- May reflect biases present in the original VNTC dataset
-- Performance varies across categories (best on sports at 98% F1-score, weakest on lifestyle at 76% F1-score)
+- Trained on specific domains (news and banking) which may have formal writing style bias
+- May reflect biases present in the original datasets
+- Performance varies significantly across categories:
+  - VNTC: Best on sports at 98% F1-score, weakest on lifestyle at 76% F1-score
+  - UTS2017_Bank: Best on TRADEMARK at 88% F1-score, many categories at 0% F1-score
 
 # 5. Future Improvements
 
@@ -109,6 +186,11 @@ Released on **2025-09-21**, the model achieves **92.33% test accuracy** and **95
 3. Add support for longer documents
 4. Implement confidence thresholds for uncertain predictions
 5. Fine-tune on domain-specific data if needed
+6. Address class imbalance issues through:
+   - Oversampling minority classes
+   - Class weight adjustments
+   - Synthetic data generation (SMOTE)
+7. Expand to more Vietnamese text domains
 
 # 6. Usage
 
@@ -118,8 +200,26 @@ pip install scikit-learn>=1.6 joblib
 ```
 
 ## 6.2 Training
+
+### VNTC Dataset (News Classification)
 ```bash
+# Default training with VNTC dataset
 uv run --no-project --with 'scikit-learn>=1.6' python train.py
+
+# With specific parameters
+uv run --no-project --with 'scikit-learn>=1.6' python train.py --model logistic --max-features 20000
+```
+
+### UTS2017_Bank Dataset (Banking Text Classification)
+```bash
+# Train with UTS2017_Bank dataset (assuming train.py is modified for UTS2017_Bank)
+python train.py --model logistic
+
+# With specific parameters
+python train.py --model logistic --max-features 20000 --ngram-min 1 --ngram-max 2
+
+# Compare multiple configurations
+python train.py --compare
 ```
 
 ## 6.3 Inference
@@ -151,16 +251,18 @@ probabilities = model.predict_proba([text])[0]
 
 1. VNTC Dataset: Hoang, Cong Duy Vu, Dien Dinh, Le Nguyen Nguyen, and Quoc Hung Ngo. (2007). A Comparative Study on Vietnamese Text Classification Methods. In Proceedings of IEEE International Conference on Research, Innovation and Vision for the Future (RIVF 2007), pp. 267-273. IEEE. DOI: 10.1109/RIVF.2007.369167
 
-2. TF-IDF (Term Frequency-Inverse Document Frequency): Salton, Gerard, and Michael J. McGill. (1983). Introduction to Modern Information Retrieval. McGraw-Hill, New York. ISBN: 978-0070544840
+2. UTS2017_Bank Dataset: Available from Hugging Face Datasets: https://huggingface.co/datasets/undertheseanlp/UTS2017_Bank
+
+3. TF-IDF (Term Frequency-Inverse Document Frequency): Salton, Gerard, and Michael J. McGill. (1983). Introduction to Modern Information Retrieval. McGraw-Hill, New York. ISBN: 978-0070544840
 
-3. Logistic Regression for Text Classification: Hastie, Trevor, Robert Tibshirani, and Jerome Friedman. (2009). The Elements of Statistical Learning: Data Mining, Inference, and Prediction (2nd ed.). Springer Series in Statistics. Springer, New York. DOI: 10.1007/978-0-387-84858-7
+4. Logistic Regression for Text Classification: Hastie, Trevor, Robert Tibshirani, and Jerome Friedman. (2009). The Elements of Statistical Learning: Data Mining, Inference, and Prediction (2nd ed.). Springer Series in Statistics. Springer, New York. DOI: 10.1007/978-0-387-84858-7
 
-4. Scikit-learn: Pedregosa, Fabian, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, Jake Vanderplas, Alexandre Passos, David Cournapeau, Matthieu Brucher, Matthieu Perrot, and Édouard Duchesnay. (2011). Scikit-learn: Machine Learning in Python. Journal of Machine Learning Research, 12(85), 2825-2830. Retrieved from https://www.jmlr.org/papers/v12/pedregosa11a.html
+5. Scikit-learn: Pedregosa, Fabian, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, Jake Vanderplas, Alexandre Passos, David Cournapeau, Matthieu Brucher, Matthieu Perrot, and Édouard Duchesnay. (2011). Scikit-learn: Machine Learning in Python. Journal of Machine Learning Research, 12(85), 2825-2830. Retrieved from https://www.jmlr.org/papers/v12/pedregosa11a.html
 
-5. N-gram Language Models: Brown, Peter F., Vincent J. Della Pietra, Peter V. deSouza, Jenifer C. Lai, and Robert L. Mercer. (1992). Class-Based n-gram Models of Natural Language. Computational Linguistics, 18(4), 467-480. Retrieved from https://aclanthology.org/J92-4003/
+6. N-gram Language Models: Brown, Peter F., Vincent J. Della Pietra, Peter V. deSouza, Jenifer C. Lai, and Robert L. Mercer. (1992). Class-Based n-gram Models of Natural Language. Computational Linguistics, 18(4), 467-480. Retrieved from https://aclanthology.org/J92-4003/
 
 # License
-Model trained on publicly available VNTC dataset. Please refer to original dataset license for usage terms.
+Model trained on publicly available VNTC and UTS2017_Bank datasets. Please refer to original dataset licenses for usage terms.
 
 # Citation
 

train.py

@@ -1,7 +1,8 @@
 #!/usr/bin/env python3
 """
-Training script for Vietnamese text classification using UTS2017_Bank dataset.
-This script trains a TF-IDF + Logistic Regression model on the UTS2017_Bank classification dataset.
+Training script for Vietnamese text classification.
+Supports both VNTC (news) and UTS2017_Bank (banking) datasets.
+This script trains a TF-IDF + Logistic Regression model on Vietnamese text classification datasets.
 """
 
 import argparse
@@ -13,6 +14,9 @@ from datetime import datetime
 
 import numpy as np
 from datasets import load_dataset
+import requests
+import zipfile
+from io import BytesIO
 from sklearn.feature_extraction.text import CountVectorizer, TfidfTransformer
 from sklearn.linear_model import LogisticRegression
 from sklearn.metrics import accuracy_score, classification_report, confusion_matrix
@@ -41,6 +45,83 @@ def setup_logging(run_name):
     return run_dir
 
 
+def load_vntc_data(split_ratio=0.2, random_state=42, n_samples=None):
+    """Load and prepare VNTC dataset
+
+    Args:
+        split_ratio: Not used for VNTC (has predefined train/test split)
+        random_state: Not used for VNTC (has predefined train/test split)
+        n_samples: Optional limit on number of samples
+
+    Returns:
+        Tuple of (X_train, y_train), (X_test, y_test)
+    """
+    print("Loading VNTC dataset...")
+
+    # Define dataset folder
+    dataset_folder = os.path.expanduser("~/.underthesea/VNTC")
+    os.makedirs(dataset_folder, exist_ok=True)
+
+    train_file = os.path.join(dataset_folder, "train.txt")
+    test_file = os.path.join(dataset_folder, "test.txt")
+
+    # Download if not exists
+    if not os.path.exists(train_file) or not os.path.exists(test_file):
+        print("Downloading VNTC dataset...")
+        url = "https://github.com/undertheseanlp/underthesea/releases/download/resources/VNTC.zip"
+
+        response = requests.get(url)
+        with zipfile.ZipFile(BytesIO(response.content)) as zip_file:
+            zip_file.extractall(dataset_folder)
+        print("Dataset downloaded and extracted.")
+
+    # Load train data
+    X_train = []
+    y_train = []
+    with open(train_file, 'r', encoding='utf-8') as f:
+        for line in f:
+            if line.strip():
+                parts = line.strip().split(' ', 1)
+                if len(parts) == 2:
+                    label = parts[0].replace('__label__', '')
+                    text = parts[1]
+                    y_train.append(label)
+                    X_train.append(text)
+
+    # Load test data
+    X_test = []
+    y_test = []
+    with open(test_file, 'r', encoding='utf-8') as f:
+        for line in f:
+            if line.strip():
+                parts = line.strip().split(' ', 1)
+                if len(parts) == 2:
+                    label = parts[0].replace('__label__', '')
+                    text = parts[1]
+                    y_test.append(label)
+                    X_test.append(text)
+
+    # Apply sample limit if specified
+    if n_samples:
+        if n_samples < len(X_train):
+            X_train = X_train[:n_samples]
+            y_train = y_train[:n_samples]
+        if n_samples < len(X_test):
+            X_test = X_test[:n_samples]
+            y_test = y_test[:n_samples]
+
+    # Convert to numpy arrays
+    X_train = np.array(X_train)
+    y_train = np.array(y_train)
+    X_test = np.array(X_test)
+    y_test = np.array(y_test)
+
+    print(f"Dataset loaded: {len(X_train)} train samples, {len(X_test)} test samples")
+    print(f"Number of unique labels: {len(set(y_train))}")
+
+    return (X_train, y_train), (X_test, y_test)
+
+
 def load_uts2017_data(split_ratio=0.2, random_state=42, n_samples=None):
     """Load and prepare UTS2017_Bank classification dataset
 
@@ -106,6 +187,7 @@ def get_available_models():
 
 
 def train_model(
+    dataset="uts2017",
     model_name="logistic",
     max_features=20000,
     ngram_range=(1, 2),
@@ -115,6 +197,7 @@ def train_model(
     """Train a single model with specified parameters
 
     Args:
+        dataset: Dataset to use ('vntc' or 'uts2017')
         model_name: Name of the model to train ('logistic' or 'svc')
         max_features: Maximum number of features for TF-IDF vectorizer
         ngram_range: N-gram range for feature extraction
@@ -139,10 +222,18 @@ def train_model(
     os.makedirs(output_folder, exist_ok=True)
 
     # Load data
-    logging.info("Loading UTS2017_Bank dataset...")
-    (X_train, y_train), (X_test, y_test) = load_uts2017_data(
-        split_ratio=split_ratio, n_samples=n_samples
-    )
+    if dataset == "vntc":
+        logging.info("Loading VNTC dataset...")
+        (X_train, y_train), (X_test, y_test) = load_vntc_data(
+            split_ratio=split_ratio, n_samples=n_samples
+        )
+        dataset_name = "VNTC"
+    else:
+        logging.info("Loading UTS2017_Bank dataset...")
+        (X_train, y_train), (X_test, y_test) = load_uts2017_data(
+            split_ratio=split_ratio, n_samples=n_samples
+        )
+        dataset_name = "UTS2017_Bank"
 
     # Get unique labels for reporting
     unique_labels = sorted(set(y_train))
@@ -167,7 +258,7 @@ def train_model(
     logging.info(f"Selected classifier: {clf_name}")
 
     # Configuration name
-    config_name = f"UTS2017_Bank_{clf_name}_feat{max_features // 1000}k_ngram{ngram_range[0]}-{ngram_range[1]}"
+    config_name = f"{dataset_name}_{clf_name}_feat{max_features // 1000}k_ngram{ngram_range[0]}-{ngram_range[1]}"
 
     logging.info("=" * 60)
     logging.info(f"Training: {config_name}")
@@ -354,23 +445,25 @@ def train_all_configurations():
     return results
 
 
-def train_notebook(model_name="logistic", max_features=20000, ngram_min=1, ngram_max=2,
+def train_notebook(dataset="uts2017", model_name="logistic", max_features=20000, ngram_min=1, ngram_max=2,
                    split_ratio=0.2, n_samples=None, compare=False):
     """
     Convenience function for training in Jupyter/Colab notebooks without argparse.
 
     Example usage:
         from train import train_notebook
-        train_notebook(model_name="logistic", max_features=20000)
+        train_notebook(dataset="vntc", model_name="logistic", max_features=20000)
     """
     if compare:
         print("Training and comparing multiple configurations...")
         return train_all_configurations()
     else:
-        print(f"Training {model_name} model on UTS2017_Bank dataset...")
+        dataset_name = "VNTC" if dataset == "vntc" else "UTS2017_Bank"
+        print(f"Training {model_name} model on {dataset_name} dataset...")
         print(f"Configuration: max_features={max_features}, ngram=({ngram_min}, {ngram_max})")
 
         return train_model(
+            dataset=dataset,
             model_name=model_name,
             max_features=max_features,
             ngram_range=(ngram_min, ngram_max),
@@ -386,7 +479,14 @@ def main():
     in_notebook = hasattr(sys, 'ps1') or 'ipykernel' in sys.modules or 'google.colab' in sys.modules
 
     parser = argparse.ArgumentParser(
-        description="Train Vietnamese text classification model on UTS2017_Bank dataset"
+        description="Train Vietnamese text classification model on VNTC or UTS2017_Bank dataset"
+    )
+    parser.add_argument(
+        "--dataset",
+        type=str,
+        choices=["vntc", "uts2017"],
+        default="uts2017",
+        help="Dataset to use for training (default: uts2017)",
     )
     parser.add_argument(
         "--model",
@@ -433,12 +533,14 @@ def main():
         print("Training and comparing multiple configurations...")
         train_all_configurations()
     else:
-        print(f"Training {args.model} model on UTS2017_Bank dataset...")
+        dataset_name = "VNTC" if args.dataset == "vntc" else "UTS2017_Bank"
+        print(f"Training {args.model} model on {dataset_name} dataset...")
         print(
             f"Configuration: max_features={args.max_features}, ngram=({args.ngram_min}, {args.ngram_max})"
         )
 
         train_model(
+            dataset=args.dataset,
             model_name=args.model,
             max_features=args.max_features,
             ngram_range=(args.ngram_min, args.ngram_max),