README.md

---
license: apache-2.0
library_name: scikit-learn
tags:
  - scikit-learn
  - sklearn
  - text-classification
  - vietnamese
  - nlp
  - pulse
  - tf-idf
  - logistic-regression
  - svc
  - support-vector-classification
  - aspect-sentiment-analysis
  - banking
  - financial-nlp
datasets:
  - undertheseanlp/UTS2017_Bank
  - ura-hcmut/vlsp2016
metrics:
  - accuracy
  - precision
  - recall
  - f1-score
model-index:
  - name: pulse-core-1
    results:
      - task:
          type: text-classification
          name: Vietnamese General Sentiment Analysis
        dataset:
          name: VLSP2016
          type: ura-hcmut/vlsp2016
        metrics:
          - type: accuracy
            value: 0.7114
            name: Test Accuracy (SVC Linear)
          - type: accuracy
            value: 0.7019
            name: Test Accuracy (Logistic Regression)
          - type: f1-score
            value: 0.713
            name: Weighted F1-Score (SVC)
          - type: f1-score
            value: 0.703
            name: Weighted F1-Score (Logistic Regression)
      - task:
          type: text-classification
          name: Vietnamese Banking Aspect Sentiment Analysis
        dataset:
          name: UTS2017_Bank
          type: undertheseanlp/UTS2017_Bank
        metrics:
          - type: accuracy
            value: 0.7172
            name: Test Accuracy (SVC)
          - type: accuracy
            value: 0.6818
            name: Test Accuracy (Logistic Regression)
          - type: precision
            value: 0.65
            name: Weighted Precision (SVC)
          - type: recall
            value: 0.72
            name: Weighted Recall (SVC)
          - type: f1-score
            value: 0.66
            name: Weighted F1-Score (SVC)
          - type: f1-score
            value: 0.66
            name: Weighted F1-Score (Logistic Regression)
language:
  - vi
pipeline_tag: text-classification
---

# Pulse Core 1 - Vietnamese Sentiment Analysis System

A comprehensive machine learning-based sentiment analysis system for Vietnamese text processing. Built on TF-IDF feature extraction pipeline combined with various machine learning algorithms, achieving **71.14% accuracy** on VLSP2016 general sentiment dataset and **71.72% accuracy** on UTS2017_Bank banking aspect sentiment dataset with Support Vector Classification (SVC).

📋 **[View Detailed System Card](https://huggingface.co/undertheseanlp/pulse_core_1/blob/main/paper/pulse_core_1_technical_report.tex)** for comprehensive model documentation, performance analysis, and limitations.

## Model Description

**Pulse Core 1** is a versatile Vietnamese sentiment analysis system that supports both general sentiment classification and specialized banking aspect sentiment analysis. The system can analyze general Vietnamese text sentiment (positive/negative/neutral) and banking-specific aspect sentiment (combining banking aspects with sentiment polarities). It's designed for Vietnamese text analysis across multiple domains, with specialized capabilities for banking customer feedback analysis and financial service categorization.

### Model Architecture

- **Algorithm**: TF-IDF + SVC/Logistic Regression Pipeline
- **Feature Extraction**: CountVectorizer with 20,000 max features
- **N-gram Support**: Unigram and bigram (1-2)
- **TF-IDF**: Transformation with IDF weighting
- **Classifier**: Support Vector Classification (SVC) / Logistic Regression with optimized parameters
- **Framework**: scikit-learn ≥1.6
- **Caching System**: Hash-based caching for efficient processing

## Supported Datasets & Categories

### VLSP2016 Dataset - General Sentiment Analysis (3 classes)

**Sentiment Categories:**
- **positive** - Positive sentiment towards products/services
- **negative** - Negative sentiment towards products/services
- **neutral** - Neutral or mixed sentiment

**Dataset Statistics:**
- Training samples: 5,100 (1,700 per class)
- Test samples: 1,050 (350 per class)
- Balanced distribution across all sentiment classes
- Domain: General product and service reviews

### UTS2017_Bank Dataset - Banking Aspect Sentiment (35 combined classes)

**Banking Aspects:**
1. **ACCOUNT** - Account services
2. **CARD** - Card services
3. **CUSTOMER_SUPPORT** - Customer support
4. **DISCOUNT** - Discount offers
5. **INTEREST_RATE** - Interest rate information
6. **INTERNET_BANKING** - Internet banking services
7. **LOAN** - Loan services
8. **MONEY_TRANSFER** - Money transfer services
9. **OTHER** - Other services
10. **PAYMENT** - Payment services
11. **PROMOTION** - Promotional offers
12. **SAVING** - Savings accounts
13. **SECURITY** - Security features
14. **TRADEMARK** - Trademark/branding

**Sentiments:**
- **positive** - Positive sentiment
- **negative** - Negative sentiment
- **neutral** - Neutral sentiment

**Combined Labels:** The model predicts combined aspect-sentiment labels in the format `<aspect>#<sentiment>`, such as:
- `CUSTOMER_SUPPORT#negative` - Negative feedback about customer support
- `LOAN#positive` - Positive opinion about loan services
- `TRADEMARK#positive` - Positive brand perception

## Installation

```bash
pip install scikit-learn>=1.6 joblib
```

## Usage

### Training the Model

#### Dataset Selection and Training

**VLSP2016 Dataset (General Sentiment Analysis):**
```bash
# Train on VLSP2016 with Logistic Regression
python train.py --dataset vlsp2016 --model logistic

# Train with SVC for better performance
python train.py --dataset vlsp2016 --model svc_linear

# Compare n-gram ranges
python train.py --dataset vlsp2016 --model svc_linear --ngram-min 1 --ngram-max 2
python train.py --dataset vlsp2016 --model svc_linear --ngram-min 1 --ngram-max 3

# Export model for deployment
python train.py --dataset vlsp2016 --model svc_linear --export-model
```

**UTS2017_Bank Dataset (Banking Aspect Sentiment Analysis):**
```bash
# Train on UTS2017_Bank (default dataset)
python train.py --dataset uts2017 --model logistic

# Train with SVC for better performance
python train.py --dataset uts2017 --model svc_linear

# With specific parameters
python train.py --dataset uts2017 --model logistic --max-features 20000 --ngram-min 1 --ngram-max 2

# Export model for deployment
python train.py --dataset uts2017 --model logistic --export-model

# Compare multiple models on specific dataset
python train.py --dataset vlsp2016 --compare-models logistic svc_linear
```

### Training from Scratch

```python
from train import train_notebook

# Train VLSP2016 general sentiment model
results = train_notebook(
    dataset="vlsp2016",
    model_name="svc_linear",
    max_features=20000,
    ngram_min=1,
    ngram_max=2,
    export_model=True
)

# Train UTS2017_Bank aspect sentiment model
results = train_notebook(
    dataset="uts2017",
    model_name="logistic",
    max_features=20000,
    ngram_min=1,
    ngram_max=2,
    export_model=True
)

# Compare multiple models on VLSP2016
comparison_results = train_notebook(
    dataset="vlsp2016",
    compare=True
)
```

## Performance Metrics

### VLSP2016 General Sentiment Analysis Performance
- **Training Accuracy**: 94.57% (SVC Linear)
- **Test Accuracy**: 71.14% (SVC Linear, 1-2 ngram) / 70.67% (SVC Linear, 1-3 ngram) / 70.19% (Logistic Regression)
- **Training Samples**: 5,100 (balanced: 1,700 per class)
- **Test Samples**: 1,050 (balanced: 350 per class)
- **Number of Classes**: 3 sentiment polarities
- **Training Time**: ~24.95 seconds (SVC) / 0.75 seconds (LR)
- **Per-Class Performance (SVC Linear)**:
  - **Positive**: 80% precision, 72% recall, 76% F1-score
  - **Negative**: 70% precision, 72% recall, 71% F1-score
  - **Neutral**: 65% precision, 69% recall, 67% F1-score
- **Key Insights**: Consistent performance across all sentiment classes due to balanced dataset
- **Optimal N-gram**: Bigrams (1-2) outperform trigrams (1-3) by 0.47 percentage points

### UTS2017_Bank Aspect Sentiment Analysis Performance
- **Training Accuracy**: 94.57% (SVC)
- **Test Accuracy**: 71.72% (SVC) / 68.18% (Logistic Regression)
- **Training Samples**: 1,581
- **Test Samples**: 396
- **Number of Classes**: 35 aspect-sentiment combinations
- **Training Time**: ~5.3 seconds (SVC) / 2.13 seconds (LR)
- **Best Performing Classes**:
  - `TRADEMARK#positive`: 90% F1-score
  - `CUSTOMER_SUPPORT#positive`: 88% F1-score
  - `LOAN#negative`: 67% F1-score (SVC improvement over LR)
  - `CUSTOMER_SUPPORT#negative`: 65% F1-score
- **Challenges**: Class imbalance affects minority aspect-sentiment combinations
- **Key Finding**: SVC shows superior category diversity compared to Logistic Regression

### Cross-Dataset Performance Analysis
- **Consistent SVC Performance**: ~71% accuracy on both 3-class (VLSP2016) and 35-class (UTS2017_Bank) tasks
- **Balance Impact**: Balanced datasets (VLSP2016) yield consistent per-class results while imbalanced datasets create performance variations
- **Training Efficiency**: Larger balanced datasets require more training time but provide stable results

## Using the Pre-trained Models

### Local Model (Vietnamese Banking Aspect Sentiment Analysis)

```python
import joblib

# Load VLSP2016 general sentiment model
general_model = joblib.load("vlsp2016_sentiment_20250929_075529.joblib")

# Load UTS2017_Bank aspect sentiment model
banking_model = joblib.load("uts2017_sentiment_20250928_131716.joblib")

# Or use inference script directly
from inference import predict_text

# General sentiment analysis
general_text = "Sản phẩm này rất tốt, tôi rất hài lòng"
prediction, confidence, top_predictions = predict_text(general_model, general_text)
print(f"General Sentiment: {prediction}")  # Expected: positive

# Banking aspect sentiment analysis
bank_text = "Lãi suất vay mua nhà hiện tại quá cao"
prediction, confidence, top_predictions = predict_text(banking_model, bank_text)
print(f"Banking Aspect-Sentiment: {prediction}")  # Expected: INTEREST_RATE#negative

print(f"Confidence: {confidence:.3f}")
print("Top 3 predictions:")
for i, (category, prob) in enumerate(top_predictions, 1):
    print(f"  {i}. {category}: {prob:.3f}")

# Example output for banking text:
# Banking Aspect-Sentiment: INTEREST_RATE#negative
# Confidence: 0.509
# Top 3 predictions:
#   1. INTEREST_RATE#negative: 0.509
#   2. LOAN#negative: 0.218
#   3. CUSTOMER_SUPPORT#negative: 0.095
```

### Using the Inference Script

```bash
# Interactive mode
python inference.py

# Single prediction
python inference.py --text "Lãi suất vay mua nhà hiện tại quá cao"

# Test with examples
python inference.py --test-examples

# List available models
python inference.py --list-models
```


## Model Parameters

- `dataset`: Dataset selection ("vlsp2016" for general sentiment, "uts2017" for banking aspect sentiment)
- `model`: Model type ("logistic", "svc_linear", "svc_rbf", "naive_bayes", "decision_tree", "random_forest", etc.)
- `max_features`: Maximum number of TF-IDF features (default: 20000)
- `ngram_min/max`: N-gram range (default: 1-2, optimal for Vietnamese)
- `split_ratio`: Train/test split ratio (default: 0.2, only used for uts2017)
- `n_samples`: Optional sample limit for quick testing
- `export_model`: Export model for deployment (creates `<dataset>_sentiment_<timestamp>.joblib`)
- `compare`: Compare multiple model configurations
- `compare_models`: Specify models to compare

## Project Management

### Cleanup Utility

The project includes a cleanup script to manage training runs:

```bash
# Preview runs that will be deleted (without exported models)
uv run python clean.py --dry-run --verbose

# Clean up runs without exported models
uv run python clean.py --yes

# Interactive cleanup with confirmation
uv run python clean.py
```

**Features:**
- Automatically identifies runs without exported model files
- Shows space that will be freed
- Dry-run mode for safe previewing
- Detailed information about each run
- Preserves runs with exported models

## Limitations

1. **Language Specificity**: Only works with Vietnamese text
2. **Domain Coverage**: Two specialized domains (general sentiment + banking aspect sentiment)
3. **Feature Limitations**: Limited to 20,000 most frequent features
4. **Class Imbalance Sensitivity**: Performance degrades significantly with imbalanced datasets (evident in UTS2017_Bank)
5. **Specific Weaknesses**:
   - **VLSP2016**: Minor performance variation between sentiment classes
   - **UTS2017_Bank**: Poor performance on minority aspect-sentiment classes due to insufficient training data
   - **N-gram Limitation**: Trigrams provide minimal improvement over bigrams while increasing computational cost
   - Banking domain aspects limited to predefined categories (account, loan, card, etc.)

## Ethical Considerations

- **Dataset Bias**: Models reflect biases present in training datasets (VLSP2016 general reviews, UTS2017_Bank banking feedback)
- **Performance Variation**: Significant performance differences between balanced (VLSP2016) and imbalanced (UTS2017_Bank) datasets
- **Domain Validation**: Should be validated on target domain before deployment
- **Class Imbalance**: Consider dataset balance when interpreting results, especially for banking aspect sentiment
- **Representation**: VLSP2016 provides more equitable performance across sentiment classes due to balanced training data

## Citation

If you use this model, please cite:

```bibtex
@misc{undertheseanlp_2025,
    author       = { Vu Anh },
    organization = { UnderTheSea NLP },
    title        = { Pulse Core 1 - Vietnamese Sentiment Analysis System },
    year         = 2025,
    url          = { https://huggingface.co/undertheseanlp/pulse_core_1 },
    doi          = { 10.57967/hf/6605 },
    publisher    = { Hugging Face }
}
```