y0un0ne/indobert-service-intent-classifier

IndoBERT Service Intent Classifier

An IndoBERT-based text classification model fine-tuned for Indonesian service intent classification.

The model classifies Indonesian user utterances into 10 service-related intent categories covering email, recommendations, takeaway services, and transportation.

It was fine-tuned from indobenchmark/indobert-base-p1 using a filtered Indonesian (id-ID) subset of the AmazonScience/massive dataset.

Model Details

• Model architecture: BERT for Sequence Classification
• Base model: indobenchmark/indobert-base-p1
• Language: Indonesian
• Task: Multi-class intent classification
• Number of classes: 10
• Maximum sequence length: 32 tokens
• Framework: PyTorch
• Library: Hugging Face Transformers

Intent Classes

The model predicts one of the following intents:

email_query
email_sendemail
recommendation_events
recommendation_locations
takeaway_order
takeaway_query
transport_query
transport_taxi
transport_ticket
transport_traffic

The classes can be grouped into four broader service domains:

Domain	Intents
Email	email_query, email_sendemail
Recommendation	recommendation_events, recommendation_locations
Takeaway	takeaway_query, takeaway_order
Transportation	transport_query, transport_taxi, transport_ticket, transport_traffic

Dataset

The model was fine-tuned using a filtered subset of the Indonesian (id-ID) portion of Amazon MASSIVE.

The original MASSIVE Indonesian dataset contains 60 intent classes. For this project, 10 service-related intents were selected.

Dataset Split

Split	Samples
Train	1,963
Validation	347
Test	488
Total	2,798

The original MASSIVE train, development, and test partitions were preserved.

The test set was not used for training or checkpoint selection.

Training

The main fine-tuning configuration was:

Parameter	Value
Training batch size	8
Evaluation batch size	8
Initial learning rate	2e-5
Weight decay	0.01
Maximum sequence length	32
Explored epochs	4
Primary selection metric	Macro F1

The best checkpoint was obtained at Epoch 3, based primarily on validation Macro F1.

Validation Performance

Epoch	Accuracy	Macro F1
1	91.64%	90.77%
2	93.37%	92.74%
3	94.24%	94.17%
4	93.08%	92.47%

Final Test Results

The selected Epoch 3 checkpoint was evaluated on the held-out test set of 488 samples.

Metric	Score
Test Loss	0.2643
Accuracy	93.24%
Macro Precision	91.78%
Macro Recall	93.33%
Macro F1	92.16%
Weighted F1	93.29%

The model correctly classified:

455 / 488 test samples

Per-Class Performance

Intent	Precision	Recall	F1
email_query	94.12%	94.12%	94.12%
email_sendemail	96.49%	96.49%	96.49%
recommendation_events	90.70%	90.70%	90.70%
recommendation_locations	78.95%	96.77%	86.96%
takeaway_order	86.96%	90.91%	88.89%
takeaway_query	100.00%	82.86%	90.62%
transport_query	95.56%	84.31%	89.58%
transport_taxi	100.00%	100.00%	100.00%
transport_ticket	100.00%	97.14%	98.55%
transport_traffic	75.00%	100.00%	85.71%

Usage

Install Dependencies

pip install torch transformers

Load the Model

import torch
from transformers import (
    AutoTokenizer,
    AutoModelForSequenceClassification
)

MODEL_NAME = "y0un0ne/indobert-service-intent-classifier"

tokenizer = AutoTokenizer.from_pretrained(MODEL_NAME)

model = AutoModelForSequenceClassification.from_pretrained(
    MODEL_NAME
)

model.eval()

Predict an Intent

text = "tolong pesankan taksi ke bandara besok pagi"

inputs = tokenizer(
    text,
    return_tensors="pt",
    truncation=True,
    max_length=32
)

with torch.no_grad():
    outputs = model(**inputs)

probabilities = torch.softmax(
    outputs.logits,
    dim=-1
)

predicted_id = probabilities.argmax(dim=-1).item()
confidence = probabilities[0, predicted_id].item()

intent = model.config.id2label[predicted_id]

print("Intent:", intent)
print("Confidence:", f"{confidence:.2%}")

Example output:

Intent: transport_taxi
Confidence: 98.67%

Example Predictions

Input	Prediction
tolong kirim email ke Andi kalau rapat besok dibatalkan	email_sendemail
bagaimana kondisi macet menuju bandara sekarang?	transport_traffic
carikan tiket pesawat murah ke Surabaya	transport_ticket
ada konser apa akhir pekan ini?	recommendation_events
tolong pesankan taksi ke bandara besok pagi	transport_taxi

Error Analysis

Most errors were concentrated among semantically overlapping intent pairs, particularly:

transport_query       <-> transport_traffic
email_query           <-> email_sendemail
takeaway_query        <-> takeaway_order
recommendation_events <-> recommendation_locations

Some test samples were also ambiguous, underspecified, or potentially affected by translation or annotation noise.

High softmax confidence should therefore not be interpreted as guaranteed correctness.

Intended Use

This model is intended for:

• Indonesian NLP experimentation
• intent classification research
• educational projects
• prototyping service-routing systems
• demonstrating IndoBERT fine-tuning workflows

The model may also be useful as a starting point for service assistants or routing systems operating on Indonesian-language queries.

Limitations

• The model only supports the 10 intent classes listed above.
• It assumes one dominant intent per input.
• It was not trained on all 60 MASSIVE intent classes.
• Several classes have overlapping semantics.
• Some training and test utterances may contain translation or annotation noise.
• Softmax confidence is not calibrated.
• The model has not been validated on production user traffic.
• Performance on Indonesian text outside the MASSIVE domain may differ.

Training Note

Training was extended incrementally through checkpoint resumption while exploring up to four epochs.

This caused learning-rate schedule adjustments between resumed training sessions.

A future controlled experiment could train the full epoch horizon in a single run with automatic best-checkpoint selection based on validation Macro F1.

Dataset Attribution

This model was fine-tuned using data derived from the Amazon MASSIVE dataset.

The project uses only the Indonesian (id-ID) subset and further filters it to 10 selected intent classes.

MASSIVE is distributed under the Creative Commons Attribution 4.0 International (CC BY 4.0) license.

Dataset:

AmazonScience/massive

Base Model

This model was fine-tuned from:

indobenchmark/indobert-base-p1

The base IndoBERT model is distributed under the MIT License.

Source Code

The complete training pipeline, notebooks, evaluation results, confusion matrix, error analysis, and local inference implementation are available on GitHub:

GitHub: https://github.com/y0un0ne/indobert-service-intent-classifier

Disclaimer

This model was developed for learning, experimentation, and portfolio purposes.

Additional robustness testing, monitoring, confidence calibration, security evaluation, and domain-specific validation are recommended before production deployment.