The model classifies Indonesian user utterances into 10 service-related intent categories covering email, recommendations, takeaway services, and transportation.

It was fine-tuned from indobenchmark/indobert-base-p1 using a filtered Indonesian (id-ID) subset of the AmazonScience/massive dataset.

Model Details

Model architecture: BERT for Sequence Classification
Base model: indobenchmark/indobert-base-p1
Language: Indonesian
Task: Multi-class intent classification
Number of classes: 10
Maximum sequence length: 32 tokens
Framework: PyTorch
Library: Hugging Face Transformers

Intent Classes

The model predicts one of the following intents:

text

email_query
email_sendemail
recommendation_events
recommendation_locations
takeaway_order
takeaway_query
transport_query
transport_taxi
transport_ticket
transport_traffic

The classes can be grouped into four broader service domains:

Domain	Intents
Email	`email_query`, `email_sendemail`
Recommendation	`recommendation_events`, `recommendation_locations`
Takeaway	`takeaway_query`, `takeaway_order`
Transportation	`transport_query`, `transport_taxi`, `transport_ticket`, `transport_traffic`

Dataset

The model was fine-tuned using a filtered subset of the Indonesian (id-ID) portion of Amazon MASSIVE.

The original MASSIVE Indonesian dataset contains 60 intent classes. For this project, 10 service-related intents were selected.

Dataset Split

Split	Samples
Train	1,963
Validation	347
Test	488
Total	2,798

The original MASSIVE train, development, and test partitions were preserved.

The test set was not used for training or checkpoint selection.

Training

The main fine-tuning configuration was:

Parameter	Value
Training batch size	8
Evaluation batch size	8
Initial learning rate	2e-5
Weight decay	0.01
Maximum sequence length	32
Explored epochs	4
Primary selection metric	Macro F1

The best checkpoint was obtained at Epoch 3, based primarily on validation Macro F1.

Validation Performance

Epoch	Accuracy	Macro F1
1	91.64%	90.77%
2	93.37%	92.74%
3	94.24%	94.17%
4	93.08%	92.47%

Final Test Results

The selected Epoch 3 checkpoint was evaluated on the held-out test set of 488 samples.

Metric	Score
Test Loss	0.2643
Accuracy	93.24%
Macro Precision	91.78%
Macro Recall	93.33%
Macro F1	92.16%
Weighted F1	93.29%

The model correctly classified:

455 / 488 test samples

Per-Class Performance

Intent	Precision	Recall	F1
`email_query`	94.12%	94.12%	94.12%
`email_sendemail`	96.49%	96.49%	96.49%
`recommendation_events`	90.70%	90.70%	90.70%
`recommendation_locations`	78.95%	96.77%	86.96%
`takeaway_order`	86.96%	90.91%	88.89%
`takeaway_query`	100.00%	82.86%	90.62%
`transport_query`	95.56%	84.31%	89.58%
`transport_taxi`	100.00%	100.00%	100.00%
`transport_ticket`	100.00%	97.14%	98.55%
`transport_traffic`	75.00%	100.00%	85.71%

Usage

Install Dependencies

pip install torch transformers

Load the Model

python

import torch
from transformers import (
    AutoTokenizer,
    AutoModelForSequenceClassification
)

MODEL_NAME = "y0un0ne/indobert-service-intent-classifier"

tokenizer = AutoTokenizer.from_pretrained(MODEL_NAME)

model = AutoModelForSequenceClassification.from_pretrained(
    MODEL_NAME
)

model.eval()

Predict an Intent

python

text = "tolong pesankan taksi ke bandara besok pagi"

inputs = tokenizer(
    text,
    return_tensors="pt",
    truncation=True,
    max_length=32
)

with torch.no_grad():
    outputs = model(**inputs)

probabilities = torch.softmax(
    outputs.logits,
    dim=-1
)

predicted_id = probabilities.argmax(dim=-1).item()
confidence = probabilities[0, predicted_id].item()

intent = model.config.id2label[predicted_id]

print("Intent:", intent)
print("Confidence:", f"{confidence:.2%}")

Example output:

text

Intent: transport_taxi
Confidence: 98.67%

Example Predictions

Input	Prediction
`tolong kirim email ke Andi kalau rapat besok dibatalkan`	`email_sendemail`
`bagaimana kondisi macet menuju bandara sekarang?`	`transport_traffic`
`carikan tiket pesawat murah ke Surabaya`	`transport_ticket`
`ada konser apa akhir pekan ini?`	`recommendation_events`
`tolong pesankan taksi ke bandara besok pagi`	`transport_taxi`

Error Analysis

Most errors were concentrated among semantically overlapping intent pairs, particularly:

text

transport_query       <-> transport_traffic
email_query           <-> email_sendemail
takeaway_query        <-> takeaway_order
recommendation_events <-> recommendation_locations

Some test samples were also ambiguous, underspecified, or potentially affected by translation or annotation noise.

High softmax confidence should therefore not be interpreted as guaranteed correctness.

Intended Use

This model is intended for:

Indonesian NLP experimentation
intent classification research
educational projects
prototyping service-routing systems
demonstrating IndoBERT fine-tuning workflows

The model may also be useful as a starting point for service assistants or routing systems operating on Indonesian-language queries.

Limitations

The model only supports the 10 intent classes listed above.
It assumes one dominant intent per input.
It was not trained on all 60 MASSIVE intent classes.
Several classes have overlapping semantics.
Some training and test utterances may contain translation or annotation noise.
Softmax confidence is not calibrated.
The model has not been validated on production user traffic.
Performance on Indonesian text outside the MASSIVE domain may differ.

Training Note

Training was extended incrementally through checkpoint resumption while exploring up to four epochs.

This caused learning-rate schedule adjustments between resumed training sessions.

A future controlled experiment could train the full epoch horizon in a single run with automatic best-checkpoint selection based on validation Macro F1.

Dataset Attribution

This model was fine-tuned using data derived from the Amazon MASSIVE dataset.

The project uses only the Indonesian (id-ID) subset and further filters it to 10 selected intent classes.

MASSIVE is distributed under the Creative Commons Attribution 4.0 International (CC BY 4.0) license.

Dataset:

AmazonScience/massive

Base Model

This model was fine-tuned from:

indobenchmark/indobert-base-p1

The base IndoBERT model is distributed under the MIT License.

Source Code

The complete training pipeline, notebooks, evaluation results, confusion matrix, error analysis, and local inference implementation are available on GitHub:

GitHub: https://github.com/y0un0ne/indobert-service-intent-classifier

Disclaimer

This model was developed for learning, experimentation, and portfolio purposes.

Additional robustness testing, monitoring, confidence calibration, security evaluation, and domain-specific validation are recommended before production deployment.

ajiyunant10/indobert-service-intent-classifier

IndoBERT Service Intent Classifier

Model Details

Intent Classes

Dataset

Dataset Split

Training

Validation Performance

Final Test Results

Per-Class Performance

Usage

Install Dependencies

Load the Model

Predict an Intent

Example Predictions

Error Analysis

Intended Use

Limitations

Training Note

Dataset Attribution

Base Model

Source Code

Disclaimer

ajiyunant10/
indobert-service-intent-classifier