🇮🇩 BPE Tokenizer — Bahasa Indonesia
A high-performance Byte Pair Encoding tokenizer built from scratch for Bahasa Indonesia
Pure Python • Zero Dependencies • 9,800+ sentences/sec • HuggingFace Compatible
📖 Overview
This tokenizer was built entirely from scratch — no SentencePiece, no HuggingFace Tokenizers library — to demonstrate the BPE algorithm and provide a tokenizer optimized for Indonesian text. It is designed as both a learning resource and a functional tokenizer for NLP tasks in Bahasa Indonesia.
✨ Key Features
Feature Description 🔧 Built from Scratch Pure Python BPE implementation with zero external dependencies 🇮🇩 Optimized for Indonesian Trained on 355+ diverse Indonesian texts across 15 categories ⚡ High Performance Greedy-by-priority algorithm — 9,800+ sentences/sec encoding speed 🤗 HuggingFace Compatible Standard file format for seamless integration 📦 Lightweight 4,000 token vocabulary, ~283 KB total size 🔄 Lossless Roundtrip Encode → Decode produces identical output
🚀 Quick Start
Installation
1 # Clone the repository
2 git clone https://huggingface.co/romizone/bpe-tokenizer-id
3
4 # No additional dependencies required!
Basic Usage
1 from bpe_tokenizer import BPETokenizer
2
3 # Load tokenizer
4 tokenizer = BPETokenizer . from_pretrained ( "./" )
5
6 # Encode text to token IDs
7 text = "Saya suka makan nasi goreng di Jakarta"
8 token_ids = tokenizer . encode ( text )
9 print ( f"Token IDs: { token_ids } " )
10
11 # Decode back to text
12 decoded = tokenizer . decode ( token_ids )
13 print ( f"Decoded: { decoded } " )
14
15 # Get tokens in string form
16 tokens = tokenizer . tokenize ( text )
17 print ( f"Tokens: { tokens } " )
🔬 Run on Google Colab
1 # Step 1: Clone repository from HuggingFace
2 !git clone https : // huggingface . co / romizone / bpe - tokenizer - id
3 % cd bpe - tokenizer - id
4
5 # Step 2: Load and use the tokenizer
6 from bpe_tokenizer import BPETokenizer
7
8 tokenizer = BPETokenizer . from_pretrained ( "./" )
9
10 # Test encoding
11 text = "Indonesia adalah negara kepulauan terbesar di dunia"
12 tokens = tokenizer . tokenize ( text )
13 ids = tokenizer . encode ( text )
14 decoded = tokenizer . decode ( ids )
15
16 print ( f"Input: { text } " )
17 print ( f"Tokens: { tokens } " )
18 print ( f"IDs: { ids } " )
19 print ( f"Decoded: { decoded } " )
💡 Tips Google Colab (klik untuk expand)
Tidak perlu install library tambahan — tokenizer ini pure Python
Untuk training ulang dengan data sendiri:
1 tokenizer = BPETokenizer ( vocab_size = 8000 )
2 tokenizer . train ( [ "teks kamu di sini" , . . . ] , min_frequency = 2 , verbose = True )
3 tokenizer . save ( "/content/my-tokenizer" )
Untuk download hasil dari Colab:
1 from google . colab import files
2 ! zip - r tokenizer . zip / content / my - tokenizer
3 files . download ( "tokenizer.zip" )
📊 Run on Kaggle
1 # Step 1: Clone repository from HuggingFace
2 !git clone https : // huggingface . co / romizone / bpe - tokenizer - id
3 import sys
4 sys . path . insert ( 0 , "/kaggle/working/bpe-tokenizer-id" )
5 % cd bpe - tokenizer - id
6
7 # Step 2: Load and use the tokenizer
8 from bpe_tokenizer import BPETokenizer
9
10 tokenizer = BPETokenizer . from_pretrained ( "./" )
11
12 # Test encoding
13 text = "Teknologi kecerdasan buatan mengubah dunia"
14 tokens = tokenizer . tokenize ( text )
15 ids = tokenizer . encode ( text )
16 decoded = tokenizer . decode ( ids )
17
18 print ( f"Input: { text } " )
19 print ( f"Tokens: { tokens } " )
20 print ( f"IDs: { ids } " )
21 print ( f"Decoded: { decoded } " )
💡 Tips Kaggle (klik untuk expand)
Kaggle working directory: /kaggle/working/
Untuk menggunakan tokenizer di notebook lain dalam session yang sama:
1 import sys
2 sys . path . insert ( 0 , "/kaggle/working/bpe-tokenizer-id" )
3 from bpe_tokenizer import BPETokenizer
Untuk menyimpan sebagai Kaggle Dataset output:
tokenizer.save("/kaggle/working/output-tokenizer")
Bisa juga install via pip jika repo sudah ada setup.py:
!pip install git+https://huggingface.co/romizone/bpe-tokenizer-id
🖥️ Run Locally
1 # Clone and use
2 git clone https://huggingface.co/romizone/bpe-tokenizer-id
3 cd bpe-tokenizer-id
4 python3 -c "
5 from bpe_tokenizer import BPETokenizer
6 tok = BPETokenizer.from_pretrained('./')
7 print(tok.tokenize('Selamat pagi Indonesia'))
8 "
Output Example
Input: "Jakarta adalah ibu kota Indonesia"
Tokens: ['jakarta', ' adalah', ' ibu', ' kota', ' indonesia']
IDs: [2063, 233, 1346, 590, 96]
Decoded: "jakarta adalah ibu kota indonesia"
Ratio: 6.6 chars/token📊 Training Details
Parameter Value Algorithm Byte Pair Encoding (BPE) Vocabulary Size 4,000 tokens Merge Rules 3,956 Training Corpus 355 curated Indonesian texts (~34,500 chars) Unique Words 1,922 Avg Token Length 6.4 characters Max Token Length 18 characters Special Tokens <PAD> <UNK> <BOS> <EOS>Case Handling Configurable (default: lowercase) Encoding Speed ~0.1 ms/sentence (9,800+ sentences/sec)
📚 Training Data Categories
The tokenizer was trained on a diverse corpus covering 15 categories of Indonesian text:
# Category 1 💻 Teknologi 2 🏛️ Indonesia & Budaya 3 💰 Ekonomi & Bisnis 4 🔬 Sains & Alam 5 🏠 Kehidupan Sehari-hari 6 ⚖️ Politik & Hukum 7 🎓 Pendidikan 8 🏥 Kesehatan
# Category 9 ⚽ Olahraga 10 🔢 Angka & Statistik 11 📜 Sejarah Indonesia 12 🍜 Kuliner & Makanan 13 🗺️ Geografi & Wisata 14 📝 Hukum & Formal 15 💬 Informal & Percakapan
🧠 How BPE Works
Step 1 Split text into characters "makan" → ['m', 'a', 'k', 'a', 'n']
Step 2 Count adjacent pairs ('a', 'n') = most frequent
Step 3 Merge most frequent pair ['m', 'a', 'k', 'an']
Step 4 Repeat until vocab target ['m', 'a', 'kan'] → ['makan']BPE produces subword tokens that efficiently represent the language:
Common words become single tokens → "indonesia" = 1 tokenRare words split into meaningful subparts → "deoksiribonukleat" = 2 tokensIndonesian morphology is naturally captured → prefixes (me-, ber-, di-) and suffixes (-kan, -an, -nya)
📁 Files
File Size Description 📄 vocab.json 72 KB Token-to-ID mapping (4,000 entries) 📄 merges.txt 39 KB BPE merge rules (3,956 rules) 📄 tokenizer.json 163 KB HuggingFace compatible format ⚙️ tokenizer_config.json < 1 KB Tokenizer configuration ⚙️ special_tokens_map.json < 1 KB Special token definitions 🐍 bpe_tokenizer.py 12 KB Source code (standalone, zero dependencies)
⚡ Performance Benchmark
Metric Value Encoding Speed 0.101 ms / sentence Throughput 9,878 sentences / sec Roundtrip Accuracy 100% (all tests passed) Save & Reload Verified (identical output)
Benchmarked on Apple Silicon with 4,000 vocab / 3,956 merge rules
🔧 Advanced Usage
Training Your Own Tokenizer
1 from bpe_tokenizer import BPETokenizer
2
3 # Initialize with custom vocab size
4 tokenizer = BPETokenizer ( vocab_size = 8000 , do_lower_case = True )
5
6 # Train on your corpus
7 texts = [ "Your Indonesian texts here..." , . . . ]
8 tokenizer . train ( texts , min_frequency = 2 , verbose = True )
9
10 # Save
11 tokenizer . save ( "./my-tokenizer" )
Loading a Saved Tokenizer
1 # Load from local directory
2 tokenizer = BPETokenizer . from_pretrained ( "./my-tokenizer" )
3
4 # Verify
5 text = "Teknologi kecerdasan buatan"
6 assert tokenizer . decode ( tokenizer . encode ( text ) ) == text . lower ( )
Deploy to HuggingFace Hub
python deploy_to_hf.py --username YOUR_USERNAME --repo-name my-tokenizer🏗️ Architecture
┌─────────────────────────────────────────────────┐
│ BPE Tokenizer │
├─────────────────────────────────────────────────┤
│ │
│ Input Text ──► Pre-tokenize (Regex) │
│ │ │
│ ▼ │
│ Character Split │
│ │ │
│ ▼ │
│ Apply Merge Rules ◄── merges.txt │
│ (Greedy-by-Priority) │
│ │ │
│ ▼ │
│ Vocab Lookup ◄────── vocab.json │
│ │ │
│ ▼ │
│ Token IDs Output │
│ │
└─────────────────────────────────────────────────┘📋 Supported Tokens
The tokenizer handles a wide range of Indonesian text:
✅ Latin characters (a-z) including rare q, x
✅ Digits (0-9) for numbers and statistics
✅ Punctuation (period, comma, hyphen, etc.)
✅ Spaces (preserved as part of tokens)
✅ Indonesian morphology (prefixes, suffixes, infixes)
✅ Loan words (technical, scientific, foreign terms)
⚠️ Limitations
Trained on a curated corpus of ~355 texts (sufficient for demo, limited for production)
Case-insensitive by default (configurable via do_lower_case parameter)
No support for accented characters (loan words like "cafe" are handled, "cafe" is not)
For production use, consider training on a larger corpus (Wikipedia ID, OSCAR, Common Crawl)
🗺️ Roadmap
👨💻 Author
Jekardah AI Lab 🇮🇩
Building AI tools for Bahasa Indonesia
📄 License
This project is licensed under the
MIT License — see the
LICENSE file for details.
MIT License
Copyright (c) 2024 Jekardah AI Lab
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files.
Made with ❤️ in Indonesia 🇮🇩
If you find this project useful, please consider giving it a ⭐ on HuggingFace!
